Nov 03 08:36:44.307412 ip-10-0-131-14 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 08:36:44.307692 ip-10-0-131-14 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 08:36:44.307798 ip-10-0-131-14 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 08:36:44.308391 ip-10-0-131-14 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 08:36:54.541838 ip-10-0-131-14 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 08:36:54.541851 ip-10-0-131-14 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4f8d1fc090704d1e8efca533f29bdc8a -- Nov 03 08:38:43.576864 ip-10-0-131-14 systemd[1]: Starting Kubernetes Kubelet... Nov 03 08:38:44.003769 ip-10-0-131-14 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:38:44.003769 ip-10-0-131-14 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 08:38:44.003769 ip-10-0-131-14 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:38:44.003769 ip-10-0-131-14 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:38:44.010919 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010793 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:44.010919 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010919 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:44.010919 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010924 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010928 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010931 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010934 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010938 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010944 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010947 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010951 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010953 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010956 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010962 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010965 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010967 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010970 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010972 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010975 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010978 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010980 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010983 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010985 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:44.011025 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010988 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010990 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010993 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010995 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.010998 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011000 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011003 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011006 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011008 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011010 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011013 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011015 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011018 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011021 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011024 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011027 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011030 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011032 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011035 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:44.011516 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011039 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011043 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011046 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011050 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011053 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011055 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011058 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011060 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011063 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011065 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011068 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011070 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011073 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011075 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011078 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011081 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011083 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011086 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011088 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:44.011972 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011091 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011093 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011096 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011098 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011101 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011103 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011106 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011110 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011113 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011116 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011118 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011121 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011123 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011126 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011129 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011131 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011134 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011139 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011142 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:44.012408 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011144 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011147 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011150 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011153 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011155 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011158 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011160 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011163 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011165 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011167 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011170 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011173 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011175 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011178 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011180 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011183 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011185 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011188 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011190 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011194 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:44.012861 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011196 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011199 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011202 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011205 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011207 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011210 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.011212 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012050 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012057 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012061 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012063 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012067 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012070 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012073 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012075 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012078 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012080 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012085 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012089 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:44.013357 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012092 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012095 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012097 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012100 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012102 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012105 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012108 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012111 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012114 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012116 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012119 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012122 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012124 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012127 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012130 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012133 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012135 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012138 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012140 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012143 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:44.013826 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012145 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012147 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012150 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012153 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012156 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012159 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012162 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012164 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012167 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012169 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012172 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012174 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012177 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012180 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012182 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012185 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012188 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012191 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012194 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:44.014309 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012196 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012199 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012201 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012204 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012206 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012210 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012214 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012217 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012220 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012223 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012225 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012228 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012230 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012232 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012235 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012237 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012240 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012242 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012245 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:44.014779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012248 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012250 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012253 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012255 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012258 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012260 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012262 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012265 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012267 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012270 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012272 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012277 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012280 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012283 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012285 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012288 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012290 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012293 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012296 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012298 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:44.015242 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012301 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012303 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012306 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012308 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012311 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012313 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012315 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012318 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012320 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012323 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012327 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012330 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012332 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012335 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012338 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012340 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012843 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012848 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012851 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012854 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:44.015872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012857 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012860 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012863 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012871 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012874 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012877 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012880 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012883 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012885 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012888 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012891 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012893 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012896 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012899 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012901 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012904 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012907 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012909 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012911 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:44.016619 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012914 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012917 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012919 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012922 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012924 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012927 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012929 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012932 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012935 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012938 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012940 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012943 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012945 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012948 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012950 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012953 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012956 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012958 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012962 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012966 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:44.017181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012971 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012974 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012977 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012979 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012982 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012984 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012987 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012990 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012993 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012995 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.012999 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013001 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013004 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013006 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013009 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013012 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013014 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013017 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013019 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:44.017891 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013022 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013025 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013028 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013032 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013036 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013039 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013042 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013045 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013047 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013050 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013053 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013055 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013058 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013062 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013065 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013067 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013069 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013072 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013075 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013077 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:44.018404 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013079 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013082 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013085 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013087 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013090 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013092 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013095 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013097 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013100 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013102 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013105 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013107 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013110 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013113 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013115 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013118 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013120 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013123 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013125 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013129 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:44.018915 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013132 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013134 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013136 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.013139 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014826 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014839 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014847 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014852 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014857 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014860 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014865 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014870 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014873 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014876 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014879 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014883 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014886 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014889 2577 flags.go:64] FLAG: --cgroup-root="" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014892 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014895 2577 flags.go:64] FLAG: --client-ca-file="" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014897 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014900 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014904 2577 flags.go:64] FLAG: --cluster-domain="" Nov 03 08:38:44.019382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014907 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014910 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014914 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014917 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014922 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014925 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014928 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014931 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014935 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014938 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014941 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014944 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014946 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014951 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014954 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014957 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014960 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014963 2577 flags.go:64] FLAG: --enable-server="true" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014966 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014971 2577 flags.go:64] FLAG: --event-burst="100" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014974 2577 flags.go:64] FLAG: --event-qps="50" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014977 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014981 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014984 2577 flags.go:64] FLAG: --eviction-hard="" Nov 03 08:38:44.019957 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014988 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014991 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014994 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.014997 2577 flags.go:64] FLAG: --eviction-soft="" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015001 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015003 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015007 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015009 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015012 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015015 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015020 2577 flags.go:64] FLAG: --feature-gates="" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015024 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015030 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015033 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015036 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015039 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015042 2577 flags.go:64] FLAG: --help="false" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015045 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015048 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015051 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015054 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015057 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015060 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 08:38:44.020566 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015063 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015066 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015069 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015072 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015075 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015078 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015081 2577 flags.go:64] FLAG: --kube-reserved="" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015083 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015086 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015089 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015092 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015095 2577 flags.go:64] FLAG: --lock-file="" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015097 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015100 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015103 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015109 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015112 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015115 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015118 2577 flags.go:64] FLAG: --logging-format="text" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015123 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015127 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015130 2577 flags.go:64] FLAG: --manifest-url="" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015133 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015138 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015141 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 08:38:44.021119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015146 2577 flags.go:64] FLAG: --max-pods="110" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015149 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015152 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015154 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015157 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015160 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015163 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015166 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015174 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015177 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015180 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015183 2577 flags.go:64] FLAG: --pod-cidr="" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015186 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015189 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015192 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015195 2577 flags.go:64] FLAG: --port="10250" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015198 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015201 2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-009fd1c5f46467c7a" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015204 2577 flags.go:64] FLAG: --qos-reserved="" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015207 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015210 2577 flags.go:64] FLAG: --register-node="true" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015213 2577 flags.go:64] FLAG: --register-with-taints="" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015217 2577 flags.go:64] FLAG: --registry-burst="10" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015219 2577 flags.go:64] FLAG: --registry-qps="5" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015222 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015225 2577 flags.go:64] FLAG: --reserved-memory="" Nov 03 08:38:44.021736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015230 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015233 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015236 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015239 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015243 2577 flags.go:64] FLAG: --runonce="false" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015246 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015250 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015253 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015256 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015259 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015262 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015265 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015270 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015273 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015276 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015279 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015281 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015285 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015288 2577 flags.go:64] FLAG: --system-cgroups="" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015291 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015296 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015299 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015302 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015306 2577 flags.go:64] FLAG: --tls-min-version="" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015309 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 08:38:44.022403 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015312 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015315 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015318 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015321 2577 flags.go:64] FLAG: --v="2" Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015326 2577 flags.go:64] FLAG: --version="false" Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015330 2577 flags.go:64] FLAG: --vmodule="" Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015334 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.015339 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017626 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017770 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017774 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017777 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017780 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017785 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017787 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017790 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017793 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017796 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017801 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017806 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:44.022999 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017809 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017812 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017815 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017817 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017820 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017822 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017825 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017828 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017831 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017833 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017836 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017839 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017841 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017845 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017851 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017854 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017857 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017859 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017862 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017866 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:44.023509 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017869 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017872 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017875 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017878 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017880 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017883 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017886 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017888 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017891 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017894 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017898 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017903 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017906 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017909 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017911 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017914 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017917 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017921 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017924 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:44.023975 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017927 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017929 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017932 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017935 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017937 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017940 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017942 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017946 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017949 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017951 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017953 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017957 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017959 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017962 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017964 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017967 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017969 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017972 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017974 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017977 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:44.024432 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017979 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017981 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017985 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017987 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017989 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017992 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017995 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.017997 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018000 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018002 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018005 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018007 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018010 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018013 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018015 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018017 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018020 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018023 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018025 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018028 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:44.024930 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018030 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018032 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018035 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018038 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018040 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018043 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018045 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018048 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018050 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018052 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018055 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018057 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018060 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018062 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018065 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.018104 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:38:44.025385 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018147 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018152 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018155 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018158 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018161 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018163 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018166 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018169 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018171 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018175 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018178 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018180 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018183 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018186 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018188 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018191 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018193 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018196 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018199 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018201 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:44.025779 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018204 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018206 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018209 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018212 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018214 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018217 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018220 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018224 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018227 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018230 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018232 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018235 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018237 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018239 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018242 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018244 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018247 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018249 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018252 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:44.026240 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018255 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018257 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018260 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018263 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018265 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018268 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018270 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018272 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018275 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018278 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018280 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018283 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018285 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018288 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018290 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018293 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018295 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018298 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018301 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018304 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:44.026706 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018306 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018309 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018311 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018314 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018316 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018318 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018321 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018323 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018336 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018339 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018341 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018344 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018346 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018349 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018351 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018354 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018357 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018359 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018362 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018364 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:44.027174 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018369 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018372 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018374 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018377 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018380 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018382 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018386 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018389 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018391 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018393 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018396 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018399 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018401 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018404 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018406 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018409 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018412 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018415 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018417 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:44.027682 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018420 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018422 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018428 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018431 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018434 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018437 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018439 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.018442 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.026171 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.026189 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.026951 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.026958 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027027 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027033 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027036 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027039 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027041 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027044 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027047 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:44.028124 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027049 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027052 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027055 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027057 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027060 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027062 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027065 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027067 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027070 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027073 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027075 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027079 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027083 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027087 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027089 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027092 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027095 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027097 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027100 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:44.028749 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027103 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027105 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027108 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027111 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027114 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027117 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027120 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027122 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027125 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027127 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027130 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027133 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027135 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027138 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027140 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027143 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027146 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027148 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027150 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:44.029193 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027153 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027155 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027158 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027160 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027163 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027166 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027168 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027170 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027173 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027175 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027178 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027180 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027183 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027186 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027188 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027191 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027194 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027197 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027200 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:44.029638 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027203 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027206 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027208 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027211 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027214 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027218 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027221 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027224 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027227 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027230 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027233 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027236 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027238 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027241 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027243 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027246 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027249 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027251 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027254 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:44.030072 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027256 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027259 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027261 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027264 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027266 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027269 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027271 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027273 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027276 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027279 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027281 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027284 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027286 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027290 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027292 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027294 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027297 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027300 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027302 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027304 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:44.030539 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027307 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027310 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027312 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.027342 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027514 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027520 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027523 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027526 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027528 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027531 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027533 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027536 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027539 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027542 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027544 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027547 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:44.031026 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027550 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027552 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027555 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027557 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027560 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027563 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027565 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027568 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027570 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027573 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027576 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027579 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027581 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027584 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027586 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027588 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027591 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027593 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027596 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027598 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:44.031397 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027601 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027603 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027606 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027610 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027614 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027617 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027620 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027622 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027625 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027627 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027631 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027633 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027636 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027638 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027640 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027643 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027645 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027648 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027650 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:44.031883 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027653 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027655 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027658 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027660 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027663 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027666 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027669 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027671 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027674 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027676 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027679 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027681 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027684 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027687 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027690 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027692 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027695 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027697 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027700 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:44.032328 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027702 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027705 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027707 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027710 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027715 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027717 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027720 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027722 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027725 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027727 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027730 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027732 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027734 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027737 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027739 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027742 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027744 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027747 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027750 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027752 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:44.032872 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027755 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027757 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027760 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027762 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027765 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027767 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027770 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027772 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027775 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027777 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027780 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027782 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027785 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027788 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027792 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:44.033345 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027795 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.027823 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027856 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027860 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027863 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027866 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027870 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027872 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027876 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027879 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027882 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027885 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027887 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027890 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027892 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027895 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:44.033721 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027898 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027901 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027903 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027906 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027908 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027911 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027914 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027916 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027919 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027921 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027924 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027928 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027931 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027933 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027936 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027938 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027941 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027945 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027947 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:44.034104 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027949 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027952 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027955 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027957 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027960 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027965 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027967 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027970 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027973 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027975 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027977 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027980 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027982 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027985 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027988 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027990 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027993 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027995 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.027998 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028000 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:44.034569 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028003 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028006 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028008 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028011 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028013 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028015 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028018 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028020 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028023 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028026 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028031 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028034 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028037 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028040 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028043 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028045 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028048 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028051 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028054 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:44.035043 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028057 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028060 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028062 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028064 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028067 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028069 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028072 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028074 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028078 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028080 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028083 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028085 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028087 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028090 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028092 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028095 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028097 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028100 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028103 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028105 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:44.035531 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028108 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028110 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028112 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028116 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028118 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028121 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028123 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028126 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028128 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028132 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028135 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028138 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028140 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:44.028143 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.028199 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.030715 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.031711 2577 server.go:1014] "Starting client certificate rotation" Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.031806 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 08:38:44.035989 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.031850 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 08:38:44.036400 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.033861 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 08:38:44.036536 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.036459 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 08:38:44.047448 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.047430 2577 log.go:25] "Validated CRI v1 runtime API" Nov 03 08:38:44.051367 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.051351 2577 log.go:25] "Validated CRI v1 image API" Nov 03 08:38:44.052715 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.052683 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 08:38:44.058824 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.058803 2577 fs.go:135] Filesystem UUIDs: map[1defff30-c349-437b-be32-5685024d7217:/dev/nvme0n1p4 42ece542-58a6-4e4d-8086-55b99d712606:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 08:38:44.058878 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.058824 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 08:38:44.064867 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.064743 2577 manager.go:217] Machine: {Timestamp:2026-11-03 08:38:44.064244385 +0000 UTC m=+0.362141405 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098745 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2baf2f86bb146adae567490cdac778 SystemUUID:ec2baf2f-86bb-146a-dae5-67490cdac778 BootID:4f8d1fc0-9070-4d1e-8efc-a533f29bdc8a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:12:e0:7e:3b:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:12:e0:7e:3b:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:fa:27:b0:c1:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 08:38:44.065527 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.065515 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 08:38:44.065670 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.065657 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 08:38:44.067745 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.067708 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 08:38:44.067891 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.067747 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-14.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 08:38:44.068522 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.068512 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 08:38:44.068560 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.068525 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 08:38:44.068560 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.068536 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 08:38:44.068560 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.068550 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 08:38:44.069802 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.069792 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 08:38:44.074512 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.074492 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 08:38:44.074657 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.074647 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 08:38:44.075462 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.075451 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 03 08:38:44.075511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.075468 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 08:38:44.075511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.075496 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 08:38:44.075511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.075507 2577 kubelet.go:405] "Adding apiserver pod source" Nov 03 08:38:44.075623 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.075522 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 08:38:44.075958 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.075943 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 08:38:44.078559 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.078537 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 08:38:44.085244 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.085225 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 08:38:44.086094 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.086081 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 08:38:44.086140 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.086113 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 08:38:44.086289 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.086278 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 08:38:44.086318 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.086294 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 08:38:44.086318 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.086302 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 08:38:44.086318 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.086307 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 08:38:44.086318 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.086313 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 08:38:44.086318 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.086319 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 08:38:44.087072 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.087063 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 08:38:44.087108 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.087074 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 08:38:44.087108 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.087081 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 08:38:44.087108 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.087093 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 08:38:44.087108 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.087107 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 08:38:44.087218 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.087116 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 08:38:44.087976 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.087967 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 08:38:44.088010 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.087977 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 08:38:44.092000 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.091988 2577 server.go:1263] "Started kubelet" Nov 03 08:38:44.092257 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.092094 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 08:38:44.092404 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.092092 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 08:38:44.092689 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.092670 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 08:38:44.092920 ip-10-0-131-14 systemd[1]: Started Kubernetes Kubelet. Nov 03 08:38:44.094427 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.094410 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 03 08:38:44.094970 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.094940 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 08:38:44.098468 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:44.098433 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 08:38:44.101152 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.101130 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 08:38:44.102423 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.102402 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 08:38:44.102594 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:44.102565 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-14.us-west-1.compute.internal\" not found" Nov 03 08:38:44.103227 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.103209 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 08:38:44.103227 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.103227 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 08:38:44.103380 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.103208 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 08:38:44.103380 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.103375 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 03 08:38:44.103441 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.103383 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 03 08:38:44.104121 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.104108 2577 factory.go:55] Registering systemd factory Nov 03 08:38:44.104153 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.104124 2577 factory.go:223] Registration of the systemd container factory successfully Nov 03 08:38:44.104373 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.104359 2577 factory.go:153] Registering CRI-O factory Nov 03 08:38:44.104373 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.104373 2577 factory.go:223] Registration of the crio container factory successfully Nov 03 08:38:44.104508 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.104433 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 08:38:44.104508 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.104456 2577 factory.go:103] Registering Raw factory Nov 03 08:38:44.104567 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.104528 2577 manager.go:1199] Started watching for new ooms in manager Nov 03 08:38:44.105790 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.105746 2577 manager.go:319] Starting recovery of all containers Nov 03 08:38:44.118034 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.118012 2577 manager.go:324] Recovery completed Nov 03 08:38:44.122342 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.122316 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:44.124025 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.123992 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:44.124091 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.124032 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:44.124091 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.124047 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:44.124684 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.124671 2577 cpu_manager.go:226] "Starting" policy="none" Nov 03 08:38:44.124684 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.124683 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 08:38:44.124751 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.124700 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 08:38:44.124751 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.124736 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:44.126726 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.126711 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:44.126780 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.126744 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:44.126780 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.126759 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:44.126780 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.126762 2577 policy_none.go:50] "Start" Nov 03 08:38:44.126780 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.126775 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 08:38:44.126899 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.126786 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 08:38:44.128723 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.128711 2577 policy_none.go:44] "Start" Nov 03 08:38:44.166946 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.166774 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 08:38:44.209539 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.193554 2577 manager.go:341] "Starting Device Plugin manager" Nov 03 08:38:44.209539 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:44.193601 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 08:38:44.209539 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.193614 2577 server.go:87] "Starting device plugin registration server" Nov 03 08:38:44.209539 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.193890 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 08:38:44.209539 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.193907 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 08:38:44.209539 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.194026 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 08:38:44.209539 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.194110 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 08:38:44.209539 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.194118 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 08:38:44.209539 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:44.195297 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 08:38:44.209539 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:44.195350 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-14.us-west-1.compute.internal\" not found" Nov 03 08:38:44.231527 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.231495 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 08:38:44.232918 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.232900 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 08:38:44.232987 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.232930 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 08:38:44.232987 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.232956 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 08:38:44.233077 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:44.232995 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 08:38:44.239858 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.239815 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-14.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 08:38:44.240346 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:44.240324 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-14.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 08:38:44.240519 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:44.239814 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-14.us-west-1.compute.internal.18e47c3642d9f7bb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-14.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-14.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 08:38:44.091959227 +0000 UTC m=+0.389856248,LastTimestamp:2026-11-03 08:38:44.091959227 +0000 UTC m=+0.389856248,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-14.us-west-1.compute.internal,}" Nov 03 08:38:44.294371 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.294087 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:44.296126 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.296099 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:44.296266 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.296134 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:44.296266 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.296144 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:44.296266 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.296171 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.308727 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.308700 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 08:38:44.308814 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.308747 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v6knc" Nov 03 08:38:44.333828 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.333762 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal"] Nov 03 08:38:44.333938 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.333868 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:44.337468 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.337450 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:44.337560 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.337497 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:44.337560 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.337509 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:44.339713 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.339701 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:44.339871 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.339853 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.339932 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.339895 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:44.340514 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.340465 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:44.340609 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.340527 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:44.340609 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.340538 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:44.340609 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.340489 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:44.340609 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.340590 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:44.340609 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.340602 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:44.342629 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.342611 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.342731 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.342635 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:44.343368 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.343352 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:44.343448 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.343376 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:44.343448 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.343385 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:44.372534 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:44.372502 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-14.us-west-1.compute.internal\" not found" node="ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.374938 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:44.374919 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-14.us-west-1.compute.internal\" not found" node="ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.395542 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.395518 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:38:44.446712 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.446685 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:38:44.495293 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.495268 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:38:44.496485 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.496460 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.496535 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:44.496501 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-14.us-west-1.compute.internal\": node \"ip-10-0-131-14.us-west-1.compute.internal\" not found" Nov 03 08:38:44.496706 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.496692 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:38:44.502744 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.502729 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.504912 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.504892 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.504966 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.504922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.504966 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.504943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6652f123a319d675474cd7afd732c542-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"6652f123a319d675474cd7afd732c542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.605576 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.605467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.605576 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.605521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.605576 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.605543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6652f123a319d675474cd7afd732c542-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"6652f123a319d675474cd7afd732c542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.605576 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.605581 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6652f123a319d675474cd7afd732c542-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"6652f123a319d675474cd7afd732c542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.605907 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.605581 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.605907 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.605642 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.625391 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.625364 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 08:38:44.625554 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.625468 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.675072 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.675031 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.678014 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.677994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:38:44.681380 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:44.681362 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 08:38:45.031700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.031669 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 08:38:45.032194 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.031882 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 08:38:45.032194 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.031890 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 08:38:45.032194 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.031871 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 08:38:45.032194 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.031844 2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aaf99b2950bda4e3eb23748958ae1964-ce6ebf1c3713135a.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-14.us-west-1.compute.internal.18e47c3644c3a6b2\": read tcp 10.0.131.14:55856->3.150.68.236:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-14.us-west-1.compute.internal.18e47c3644c3a6b2 default 4780 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-14.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-14.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-14.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 08:38:44 +0000 UTC,LastTimestamp:2026-11-03 08:38:44.296148251 +0000 UTC m=+0.594045272,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-14.us-west-1.compute.internal,}" Nov 03 08:38:45.075806 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.075780 2577 apiserver.go:52] "Watching apiserver" Nov 03 08:38:45.101719 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.101691 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 08:38:45.313424 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.313306 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 08:33:45 +0000 UTC" deadline="2028-04-25 06:33:46.003052522 +0000 UTC" Nov 03 08:38:45.313424 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.313363 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12933h55m0.68969423s" Nov 03 08:38:45.345037 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.344989 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 08:38:45.345348 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.345328 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-q7vbp","openshift-network-operator/iptables-alerter-zk7xq","openshift-ovn-kubernetes/ovnkube-node-fnh7l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z","openshift-dns/node-resolver-2xrz9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-9qzgm","kube-system/global-pull-secret-syncer-c9fnb","kube-system/konnectivity-agent-hp76w","openshift-image-registry/node-ca-wkb5w","openshift-network-diagnostics/network-check-target-4cbq5","openshift-multus/multus-8hwgr","openshift-multus/multus-additional-cni-plugins-4chsq"] Nov 03 08:38:45.347816 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.347791 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:45.347918 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.347860 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:38:45.349789 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.349770 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:38:45.352567 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.352136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.354594 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.354574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.356843 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.356819 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:38:45.358902 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.358886 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.361004 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.360992 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:45.361060 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.361044 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:38:45.363242 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.363223 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:45.365364 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.365347 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:38:45.367703 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.367687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:45.367765 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.367739 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:38:45.369889 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.369877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.372128 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.372111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.404460 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.404432 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 08:38:45.404460 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.404448 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:38:45.404616 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.404504 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 08:38:45.404616 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.404508 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sntnv\"" Nov 03 08:38:45.404912 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.404898 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 08:38:45.405720 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.405700 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 08:38:45.405950 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.405927 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 08:38:45.405950 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.405936 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 08:38:45.406087 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.406036 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cwzpz\"" Nov 03 08:38:45.406261 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.406243 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 08:38:45.406364 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.406271 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 08:38:45.406427 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.406402 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 08:38:45.407680 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.407666 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v9lhl\"" Nov 03 08:38:45.407737 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.407667 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 08:38:45.410273 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcv8t\" (UniqueName: \"kubernetes.io/projected/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-kube-api-access-tcv8t\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.410387 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-bin\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.410387 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410293 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-run\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.410387 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-multus\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.410387 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410365 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-hostroot\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.410557 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:45.410557 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-var-lib-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.410557 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410506 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.410557 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-env-overrides\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.410667 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2fcj\" (UniqueName: \"kubernetes.io/projected/83c6ab8c-6c62-4163-a789-1edb75af2ad9-kube-api-access-h2fcj\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.410667 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410623 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.410727 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-kubelet\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.410727 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.410909 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gnnf\" (UniqueName: \"kubernetes.io/projected/63810bfe-e55b-4524-bde4-8f072b4b63a3-kube-api-access-4gnnf\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:45.410909 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-log-socket\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.410909 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-config\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.410909 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410822 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-modprobe-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.410909 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-dbus\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:45.411153 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:45.411153 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410937 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-device-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.411153 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.410964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3a0d3db-7b81-4e10-b240-b665c083bda2-serviceca\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:38:45.411153 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v85s\" (UniqueName: \"kubernetes.io/projected/39df63cd-3b12-4930-8a23-04ed67d961a8-kube-api-access-8v85s\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.411153 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-slash\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.411153 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.411153 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-tmp\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.411153 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1330da31-fcb6-4c7b-9db7-0d4d96262234-konnectivity-ca\") pod \"konnectivity-agent-hp76w\" (UID: \"1330da31-fcb6-4c7b-9db7-0d4d96262234\") " pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:45.411511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.411511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411223 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xccvh\" (UniqueName: \"kubernetes.io/projected/07f9e882-e987-44b0-a688-45e3c9abec5c-kube-api-access-xccvh\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:38:45.411511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-systemd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.411511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-tmp-dir\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:38:45.411511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-etc-selinux\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.411511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-k8s-cni-cncf-io\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.411511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.411511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411423 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-ovn\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.411511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-kubernetes\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.411813 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhpmt\" (UniqueName: \"kubernetes.io/projected/bfbee40d-3926-4eae-babe-e598bfae8098-kube-api-access-bhpmt\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.411813 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-conf-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.411813 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-etc-kubernetes\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.411813 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07f9e882-e987-44b0-a688-45e3c9abec5c-host-slash\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:38:45.411813 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th46k\" (UniqueName: \"kubernetes.io/projected/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-kube-api-access-th46k\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:38:45.411813 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-sys\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.411813 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-registration-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.411813 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-cnibin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.411813 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-bin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.412046 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-binary-copy\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.412046 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07f9e882-e987-44b0-a688-45e3c9abec5c-iptables-alerter-script\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:38:45.412046 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-hosts-file\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:38:45.412046 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-daemon-config\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.412046 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-multus-certs\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.412046 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-netns\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.412046 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.411992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-netd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.412046 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412024 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.412046 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-conf\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.412358 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-tuned\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.412358 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412095 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cnibin\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.412358 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j55t5\" (UniqueName: \"kubernetes.io/projected/1959beea-2daf-4e4a-a412-9bc2d35fc277-kube-api-access-j55t5\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.412358 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412190 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1330da31-fcb6-4c7b-9db7-0d4d96262234-agent-certs\") pod \"konnectivity-agent-hp76w\" (UID: \"1330da31-fcb6-4c7b-9db7-0d4d96262234\") " pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:45.412358 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-sys-fs\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.412358 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qdq7\" (UniqueName: \"kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7\") pod \"network-check-target-4cbq5\" (UID: \"f70226b0-5f4c-465c-9ec7-7ceee1bb376f\") " pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:45.412358 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412282 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-node-log\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.412358 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412302 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-script-lib\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.412358 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-var-lib-kubelet\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.412358 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-socket-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-socket-dir-parent\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412415 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-kubelet\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-systemd-units\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412498 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovn-node-metrics-cert\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3a0d3db-7b81-4e10-b240-b665c083bda2-host\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-system-cni-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-etc-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-systemd\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-lib-modules\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-system-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-netns\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.412741 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412742 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysconfig\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.413148 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-host\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.413148 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412805 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rc2j\" (UniqueName: \"kubernetes.io/projected/d3a0d3db-7b81-4e10-b240-b665c083bda2-kube-api-access-7rc2j\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:38:45.413148 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-cni-binary-copy\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.413148 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.412856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-kubelet-config\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:45.449087 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.448962 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 08:38:45.449087 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.448992 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 08:38:45.449087 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.448996 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rnxpp\"" Nov 03 08:38:45.449087 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.449007 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:38:45.449087 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.449057 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 08:38:45.449087 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.449072 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 08:38:45.451762 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.451748 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 08:38:45.499847 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.499810 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 08:38:45.499977 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.499911 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q84wj" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.505383 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.505795 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.505842 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.505879 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.505965 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.506022 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hvj2z\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.506080 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.506159 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.506235 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.506306 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.506625 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.506642 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8lmql\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.506745 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.506757 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jjcxd\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.506829 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jnk2x\"" Nov 03 08:38:45.507075 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.506977 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tps59\"" Nov 03 08:38:45.514076 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.514194 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xccvh\" (UniqueName: \"kubernetes.io/projected/07f9e882-e987-44b0-a688-45e3c9abec5c-kube-api-access-xccvh\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:38:45.514194 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-systemd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.514194 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-tmp-dir\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:38:45.514194 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-etc-selinux\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.514194 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514185 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-systemd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.514410 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-k8s-cni-cncf-io\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.514410 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514249 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-etc-selinux\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.514410 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.514410 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-ovn\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.514410 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514303 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.514410 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514297 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-k8s-cni-cncf-io\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.514410 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514346 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-ovn\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.514410 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-kubernetes\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.514410 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514367 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-kubernetes\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.514410 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514385 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhpmt\" (UniqueName: \"kubernetes.io/projected/bfbee40d-3926-4eae-babe-e598bfae8098-kube-api-access-bhpmt\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514410 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-conf-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514443 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-tmp-dir\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-etc-kubernetes\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07f9e882-e987-44b0-a688-45e3c9abec5c-host-slash\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514524 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-etc-kubernetes\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514527 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-conf-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th46k\" (UniqueName: \"kubernetes.io/projected/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-kube-api-access-th46k\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514564 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07f9e882-e987-44b0-a688-45e3c9abec5c-host-slash\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-sys\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-registration-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514603 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-sys\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-cnibin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-bin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514649 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-registration-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-binary-copy\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07f9e882-e987-44b0-a688-45e3c9abec5c-iptables-alerter-script\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514687 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-cnibin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.514936 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514687 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-bin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-hosts-file\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-daemon-config\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-multus-certs\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-netns\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-netd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514836 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-hosts-file\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-conf\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514880 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-netns\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514924 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-multus-certs\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.514968 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-netd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515008 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-conf\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-tuned\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515053 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cnibin\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515096 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cnibin\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515098 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.515710 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j55t5\" (UniqueName: \"kubernetes.io/projected/1959beea-2daf-4e4a-a412-9bc2d35fc277-kube-api-access-j55t5\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1330da31-fcb6-4c7b-9db7-0d4d96262234-agent-certs\") pod \"konnectivity-agent-hp76w\" (UID: \"1330da31-fcb6-4c7b-9db7-0d4d96262234\") " pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515152 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-binary-copy\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-sys-fs\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdq7\" (UniqueName: \"kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7\") pod \"network-check-target-4cbq5\" (UID: \"f70226b0-5f4c-465c-9ec7-7ceee1bb376f\") " pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-node-log\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-script-lib\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-var-lib-kubelet\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515276 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07f9e882-e987-44b0-a688-45e3c9abec5c-iptables-alerter-script\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515287 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-socket-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515334 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-daemon-config\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515338 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-node-log\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515375 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-var-lib-kubelet\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515393 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-sys-fs\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515380 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-socket-dir-parent\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-kubelet\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.516510 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515510 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-socket-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-systemd-units\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515547 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-socket-dir-parent\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515553 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-kubelet\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515586 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovn-node-metrics-cert\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3a0d3db-7b81-4e10-b240-b665c083bda2-host\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515643 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-systemd-units\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515656 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3a0d3db-7b81-4e10-b240-b665c083bda2-host\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-system-cni-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-etc-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515762 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-systemd\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515734 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-system-cni-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-lib-modules\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515808 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-system-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-netns\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.517314 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysconfig\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515854 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515868 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-host\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515906 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-systemd\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515909 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-etc-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515934 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-lib-modules\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rc2j\" (UniqueName: \"kubernetes.io/projected/d3a0d3db-7b81-4e10-b240-b665c083bda2-kube-api-access-7rc2j\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515980 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-netns\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515950 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-script-lib\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515985 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-system-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-cni-binary-copy\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516024 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-host\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.515954 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysconfig\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-kubelet-config\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcv8t\" (UniqueName: \"kubernetes.io/projected/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-kube-api-access-tcv8t\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-bin\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-run\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-multus\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.517900 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-hostroot\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516182 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-bin\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-var-lib-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.516259 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-env-overrides\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2fcj\" (UniqueName: \"kubernetes.io/projected/83c6ab8c-6c62-4163-a789-1edb75af2ad9-kube-api-access-h2fcj\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.516317 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs podName:63810bfe-e55b-4524-bde4-8f072b4b63a3 nodeName:}" failed. No retries permitted until 2026-11-03 08:38:46.016298985 +0000 UTC m=+2.314196014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs") pod "network-metrics-daemon-q7vbp" (UID: "63810bfe-e55b-4524-bde4-8f072b4b63a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-kubelet\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516415 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-hostroot\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516465 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-run\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516528 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-multus\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516344 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-kubelet-config\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gnnf\" (UniqueName: \"kubernetes.io/projected/63810bfe-e55b-4524-bde4-8f072b4b63a3-kube-api-access-4gnnf\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:45.518922 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516622 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-log-socket\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-config\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516660 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-var-lib-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-modprobe-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-dbus\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-device-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-cni-binary-copy\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3a0d3db-7b81-4e10-b240-b665c083bda2-serviceca\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v85s\" (UniqueName: \"kubernetes.io/projected/39df63cd-3b12-4930-8a23-04ed67d961a8-kube-api-access-8v85s\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-slash\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-tmp\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1330da31-fcb6-4c7b-9db7-0d4d96262234-konnectivity-ca\") pod \"konnectivity-agent-hp76w\" (UID: \"1330da31-fcb6-4c7b-9db7-0d4d96262234\") " pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.516912 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.516958 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret podName:25b42294-fcbd-4bc8-8652-9d4d7e762e3d nodeName:}" failed. No retries permitted until 2026-11-03 08:38:46.016942774 +0000 UTC m=+2.314839785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret") pod "global-pull-secret-syncer-c9fnb" (UID: "25b42294-fcbd-4bc8-8652-9d4d7e762e3d") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:45.519778 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.516588 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.517014 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-log-socket\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.517039 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-env-overrides\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.517108 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-kubelet\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.517221 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-modprobe-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.517275 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.517342 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-device-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.517282 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-dbus\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.517596 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.517645 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-slash\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.518775 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-config\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.518803 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3a0d3db-7b81-4e10-b240-b665c083bda2-serviceca\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.518847 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1330da31-fcb6-4c7b-9db7-0d4d96262234-konnectivity-ca\") pod \"konnectivity-agent-hp76w\" (UID: \"1330da31-fcb6-4c7b-9db7-0d4d96262234\") " pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.519675 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-tuned\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.519776 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovn-node-metrics-cert\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.519956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1330da31-fcb6-4c7b-9db7-0d4d96262234-agent-certs\") pod \"konnectivity-agent-hp76w\" (UID: \"1330da31-fcb6-4c7b-9db7-0d4d96262234\") " pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:45.520414 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.520251 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-tmp\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.573260 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.572888 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:38:45.573260 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.572925 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:38:45.573260 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.572942 2577 projected.go:196] Error preparing data for projected volume kube-api-access-4qdq7 for pod openshift-network-diagnostics/network-check-target-4cbq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:45.573260 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:45.573018 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7 podName:f70226b0-5f4c-465c-9ec7-7ceee1bb376f nodeName:}" failed. No retries permitted until 2026-11-03 08:38:46.072996171 +0000 UTC m=+2.370893199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4qdq7" (UniqueName: "kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7") pod "network-check-target-4cbq5" (UID: "f70226b0-5f4c-465c-9ec7-7ceee1bb376f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:45.576568 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.576151 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xccvh\" (UniqueName: \"kubernetes.io/projected/07f9e882-e987-44b0-a688-45e3c9abec5c-kube-api-access-xccvh\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:38:45.576568 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.576215 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th46k\" (UniqueName: \"kubernetes.io/projected/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-kube-api-access-th46k\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:38:45.576568 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.576327 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhpmt\" (UniqueName: \"kubernetes.io/projected/bfbee40d-3926-4eae-babe-e598bfae8098-kube-api-access-bhpmt\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.576851 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.576615 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rc2j\" (UniqueName: \"kubernetes.io/projected/d3a0d3db-7b81-4e10-b240-b665c083bda2-kube-api-access-7rc2j\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:38:45.576851 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.576688 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j55t5\" (UniqueName: \"kubernetes.io/projected/1959beea-2daf-4e4a-a412-9bc2d35fc277-kube-api-access-j55t5\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.577232 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.577204 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2fcj\" (UniqueName: \"kubernetes.io/projected/83c6ab8c-6c62-4163-a789-1edb75af2ad9-kube-api-access-h2fcj\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.577343 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.577205 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcv8t\" (UniqueName: \"kubernetes.io/projected/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-kube-api-access-tcv8t\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.577411 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.577381 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v85s\" (UniqueName: \"kubernetes.io/projected/39df63cd-3b12-4930-8a23-04ed67d961a8-kube-api-access-8v85s\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.578307 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.578282 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gnnf\" (UniqueName: \"kubernetes.io/projected/63810bfe-e55b-4524-bde4-8f072b4b63a3-kube-api-access-4gnnf\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:45.683978 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.683944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:38:45.711374 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.711330 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:38:45.716871 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.716421 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:38:45.727780 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.727726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:38:45.741235 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:45.741191 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b188283236722665eb092553b8ae03f.slice/crio-04a07db94e82003f36e5228103773f0fb3c2aff82c8d90a8da63b3e4fde9910d WatchSource:0}: Error finding container 04a07db94e82003f36e5228103773f0fb3c2aff82c8d90a8da63b3e4fde9910d: Status 404 returned error can't find the container with id 04a07db94e82003f36e5228103773f0fb3c2aff82c8d90a8da63b3e4fde9910d Nov 03 08:38:45.741798 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:45.741775 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbee40d_3926_4eae_babe_e598bfae8098.slice/crio-528a230da6237638550e125b0400d35a8dd30bfb70480c4b4c93a6ab9040d71d WatchSource:0}: Error finding container 528a230da6237638550e125b0400d35a8dd30bfb70480c4b4c93a6ab9040d71d: Status 404 returned error can't find the container with id 528a230da6237638550e125b0400d35a8dd30bfb70480c4b4c93a6ab9040d71d Nov 03 08:38:45.742582 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:45.742524 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1959beea_2daf_4e4a_a412_9bc2d35fc277.slice/crio-c4a924bac8d7adf1065970ff2db52bcea8d3fb3a8206e908d30aa0af07e95b08 WatchSource:0}: Error finding container c4a924bac8d7adf1065970ff2db52bcea8d3fb3a8206e908d30aa0af07e95b08: Status 404 returned error can't find the container with id c4a924bac8d7adf1065970ff2db52bcea8d3fb3a8206e908d30aa0af07e95b08 Nov 03 08:38:45.743935 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:45.743915 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d5c7ae_f8aa_4300_b5a6_c76273b66b3c.slice/crio-db87f4689840a00da5b1ae876f69e4a7be3d81d86d491189367e9bfc7e5d29db WatchSource:0}: Error finding container db87f4689840a00da5b1ae876f69e4a7be3d81d86d491189367e9bfc7e5d29db: Status 404 returned error can't find the container with id db87f4689840a00da5b1ae876f69e4a7be3d81d86d491189367e9bfc7e5d29db Nov 03 08:38:45.744352 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:45.744330 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6652f123a319d675474cd7afd732c542.slice/crio-3c55c96a34b1d5c44d14769afe45feb77800e5a27121da34071bb9e1ec64763d WatchSource:0}: Error finding container 3c55c96a34b1d5c44d14769afe45feb77800e5a27121da34071bb9e1ec64763d: Status 404 returned error can't find the container with id 3c55c96a34b1d5c44d14769afe45feb77800e5a27121da34071bb9e1ec64763d Nov 03 08:38:45.748539 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.748518 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:38:45.748998 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.748979 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:38:45.760553 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:45.760521 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c6ab8c_6c62_4163_a789_1edb75af2ad9.slice/crio-5999b4659d91e5c7e416e78bbd75e84f41ac6ae7e81f3c3ec21d7a4fe25ed958 WatchSource:0}: Error finding container 5999b4659d91e5c7e416e78bbd75e84f41ac6ae7e81f3c3ec21d7a4fe25ed958: Status 404 returned error can't find the container with id 5999b4659d91e5c7e416e78bbd75e84f41ac6ae7e81f3c3ec21d7a4fe25ed958 Nov 03 08:38:45.772302 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.772274 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:45.782342 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:45.782309 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1330da31_fcb6_4c7b_9db7_0d4d96262234.slice/crio-dc063fcc6f21eebf842e0c3a0c9789bf43cad02feebac8574b72e2cad89eda17 WatchSource:0}: Error finding container dc063fcc6f21eebf842e0c3a0c9789bf43cad02feebac8574b72e2cad89eda17: Status 404 returned error can't find the container with id dc063fcc6f21eebf842e0c3a0c9789bf43cad02feebac8574b72e2cad89eda17 Nov 03 08:38:45.784576 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.784543 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:38:45.794181 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:45.794156 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a0d3db_7b81_4e10_b240_b665c083bda2.slice/crio-e134794b81712988574b0bd790017c2d1f504c5c107d179a506c9c242314eb22 WatchSource:0}: Error finding container e134794b81712988574b0bd790017c2d1f504c5c107d179a506c9c242314eb22: Status 404 returned error can't find the container with id e134794b81712988574b0bd790017c2d1f504c5c107d179a506c9c242314eb22 Nov 03 08:38:45.802629 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.802606 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8hwgr" Nov 03 08:38:45.812428 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:45.812405 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39df63cd_3b12_4930_8a23_04ed67d961a8.slice/crio-fa9b681cd0ab0633576f0557e9d9631883b42c814c3526bf0b97f88c954c92c6 WatchSource:0}: Error finding container fa9b681cd0ab0633576f0557e9d9631883b42c814c3526bf0b97f88c954c92c6: Status 404 returned error can't find the container with id fa9b681cd0ab0633576f0557e9d9631883b42c814c3526bf0b97f88c954c92c6 Nov 03 08:38:45.822646 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.822619 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:38:45.832863 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:45.832830 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0893a4fe_c6b7_42e5_8e5c_8d179dfefedc.slice/crio-9d2b9473c11f04c4cbc7bba86a228f5f5c01ec24ac2bc0159324462bb953b204 WatchSource:0}: Error finding container 9d2b9473c11f04c4cbc7bba86a228f5f5c01ec24ac2bc0159324462bb953b204: Status 404 returned error can't find the container with id 9d2b9473c11f04c4cbc7bba86a228f5f5c01ec24ac2bc0159324462bb953b204 Nov 03 08:38:45.994582 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:45.994551 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:38:46.021199 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.021162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:46.021381 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.021206 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:46.021381 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:46.021322 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:46.021381 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:46.021331 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:46.021381 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:46.021378 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret podName:25b42294-fcbd-4bc8-8652-9d4d7e762e3d nodeName:}" failed. No retries permitted until 2026-11-03 08:38:47.021364077 +0000 UTC m=+3.319261101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret") pod "global-pull-secret-syncer-c9fnb" (UID: "25b42294-fcbd-4bc8-8652-9d4d7e762e3d") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:46.021580 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:46.021393 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs podName:63810bfe-e55b-4524-bde4-8f072b4b63a3 nodeName:}" failed. No retries permitted until 2026-11-03 08:38:47.021386852 +0000 UTC m=+3.319283861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs") pod "network-metrics-daemon-q7vbp" (UID: "63810bfe-e55b-4524-bde4-8f072b4b63a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:46.074072 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.073991 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:38:46.121841 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.121794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdq7\" (UniqueName: \"kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7\") pod \"network-check-target-4cbq5\" (UID: \"f70226b0-5f4c-465c-9ec7-7ceee1bb376f\") " pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:46.122017 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:46.121977 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:38:46.122017 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:46.121995 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:38:46.122017 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:46.122004 2577 projected.go:196] Error preparing data for projected volume kube-api-access-4qdq7 for pod openshift-network-diagnostics/network-check-target-4cbq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:46.122114 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:46.122061 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7 podName:f70226b0-5f4c-465c-9ec7-7ceee1bb376f nodeName:}" failed. No retries permitted until 2026-11-03 08:38:47.122045078 +0000 UTC m=+3.419942089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qdq7" (UniqueName: "kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7") pod "network-check-target-4cbq5" (UID: "f70226b0-5f4c-465c-9ec7-7ceee1bb376f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:46.238095 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.238032 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wkb5w" event={"ID":"d3a0d3db-7b81-4e10-b240-b665c083bda2","Type":"ContainerStarted","Data":"e134794b81712988574b0bd790017c2d1f504c5c107d179a506c9c242314eb22"} Nov 03 08:38:46.239190 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.239165 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2xrz9" event={"ID":"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c","Type":"ContainerStarted","Data":"db87f4689840a00da5b1ae876f69e4a7be3d81d86d491189367e9bfc7e5d29db"} Nov 03 08:38:46.240294 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.240272 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" event={"ID":"1959beea-2daf-4e4a-a412-9bc2d35fc277","Type":"ContainerStarted","Data":"c4a924bac8d7adf1065970ff2db52bcea8d3fb3a8206e908d30aa0af07e95b08"} Nov 03 08:38:46.241336 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.241316 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" event={"ID":"0b188283236722665eb092553b8ae03f","Type":"ContainerStarted","Data":"04a07db94e82003f36e5228103773f0fb3c2aff82c8d90a8da63b3e4fde9910d"} Nov 03 08:38:46.242432 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.242404 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4chsq" event={"ID":"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc","Type":"ContainerStarted","Data":"9d2b9473c11f04c4cbc7bba86a228f5f5c01ec24ac2bc0159324462bb953b204"} Nov 03 08:38:46.243405 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.243384 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8hwgr" event={"ID":"39df63cd-3b12-4930-8a23-04ed67d961a8","Type":"ContainerStarted","Data":"fa9b681cd0ab0633576f0557e9d9631883b42c814c3526bf0b97f88c954c92c6"} Nov 03 08:38:46.244531 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.244510 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hp76w" event={"ID":"1330da31-fcb6-4c7b-9db7-0d4d96262234","Type":"ContainerStarted","Data":"dc063fcc6f21eebf842e0c3a0c9789bf43cad02feebac8574b72e2cad89eda17"} Nov 03 08:38:46.246183 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.246153 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" event={"ID":"83c6ab8c-6c62-4163-a789-1edb75af2ad9","Type":"ContainerStarted","Data":"5999b4659d91e5c7e416e78bbd75e84f41ac6ae7e81f3c3ec21d7a4fe25ed958"} Nov 03 08:38:46.247726 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.247702 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" event={"ID":"bfbee40d-3926-4eae-babe-e598bfae8098","Type":"ContainerStarted","Data":"528a230da6237638550e125b0400d35a8dd30bfb70480c4b4c93a6ab9040d71d"} Nov 03 08:38:46.248965 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.248932 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" event={"ID":"6652f123a319d675474cd7afd732c542","Type":"ContainerStarted","Data":"3c55c96a34b1d5c44d14769afe45feb77800e5a27121da34071bb9e1ec64763d"} Nov 03 08:38:46.325342 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.325247 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:38:46.503532 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.503449 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 08:33:46 +0000 UTC" deadline="2028-08-15 18:30:30.91958885 +0000 UTC" Nov 03 08:38:46.503532 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:46.503520 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15633h51m44.416072989s" Nov 03 08:38:46.585697 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:38:46.585665 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f9e882_e987_44b0_a688_45e3c9abec5c.slice/crio-96375632bb6d392b66b23f154fdb8ad49256df6c901e8369bc81a63e49fbcbf0 WatchSource:0}: Error finding container 96375632bb6d392b66b23f154fdb8ad49256df6c901e8369bc81a63e49fbcbf0: Status 404 returned error can't find the container with id 96375632bb6d392b66b23f154fdb8ad49256df6c901e8369bc81a63e49fbcbf0 Nov 03 08:38:47.030942 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:47.030896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:47.031285 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:47.030969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:47.031285 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:47.031169 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:47.031285 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:47.031235 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret podName:25b42294-fcbd-4bc8-8652-9d4d7e762e3d nodeName:}" failed. No retries permitted until 2026-11-03 08:38:49.031213789 +0000 UTC m=+5.329110814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret") pod "global-pull-secret-syncer-c9fnb" (UID: "25b42294-fcbd-4bc8-8652-9d4d7e762e3d") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:47.031531 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:47.031303 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:47.031531 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:47.031338 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs podName:63810bfe-e55b-4524-bde4-8f072b4b63a3 nodeName:}" failed. No retries permitted until 2026-11-03 08:38:49.031325328 +0000 UTC m=+5.329222352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs") pod "network-metrics-daemon-q7vbp" (UID: "63810bfe-e55b-4524-bde4-8f072b4b63a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:47.132248 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:47.132200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdq7\" (UniqueName: \"kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7\") pod \"network-check-target-4cbq5\" (UID: \"f70226b0-5f4c-465c-9ec7-7ceee1bb376f\") " pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:47.132829 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:47.132386 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:38:47.132829 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:47.132406 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:38:47.132829 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:47.132421 2577 projected.go:196] Error preparing data for projected volume kube-api-access-4qdq7 for pod openshift-network-diagnostics/network-check-target-4cbq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:47.132829 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:47.132486 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7 podName:f70226b0-5f4c-465c-9ec7-7ceee1bb376f nodeName:}" failed. No retries permitted until 2026-11-03 08:38:49.132467082 +0000 UTC m=+5.430364108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qdq7" (UniqueName: "kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7") pod "network-check-target-4cbq5" (UID: "f70226b0-5f4c-465c-9ec7-7ceee1bb376f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:47.233429 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:47.233389 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:47.233656 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:47.233555 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:38:47.233743 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:47.233656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:47.233804 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:47.233760 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:38:47.233863 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:47.233828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:47.233914 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:47.233902 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:38:47.280694 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:47.280571 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zk7xq" event={"ID":"07f9e882-e987-44b0-a688-45e3c9abec5c","Type":"ContainerStarted","Data":"96375632bb6d392b66b23f154fdb8ad49256df6c901e8369bc81a63e49fbcbf0"} Nov 03 08:38:47.504329 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:47.504274 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 08:33:46 +0000 UTC" deadline="2028-05-13 04:48:26.968163732 +0000 UTC" Nov 03 08:38:47.504329 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:47.504327 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13364h9m39.463842168s" Nov 03 08:38:49.050654 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:49.050157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:49.050654 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:49.050225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:49.050654 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:49.050397 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:49.050654 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:49.050474 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret podName:25b42294-fcbd-4bc8-8652-9d4d7e762e3d nodeName:}" failed. No retries permitted until 2026-11-03 08:38:53.050454929 +0000 UTC m=+9.348351956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret") pod "global-pull-secret-syncer-c9fnb" (UID: "25b42294-fcbd-4bc8-8652-9d4d7e762e3d") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:49.050654 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:49.050569 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:49.050654 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:49.050610 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs podName:63810bfe-e55b-4524-bde4-8f072b4b63a3 nodeName:}" failed. No retries permitted until 2026-11-03 08:38:53.050596432 +0000 UTC m=+9.348493446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs") pod "network-metrics-daemon-q7vbp" (UID: "63810bfe-e55b-4524-bde4-8f072b4b63a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:49.151937 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:49.151558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdq7\" (UniqueName: \"kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7\") pod \"network-check-target-4cbq5\" (UID: \"f70226b0-5f4c-465c-9ec7-7ceee1bb376f\") " pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:49.151937 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:49.151797 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:38:49.151937 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:49.151819 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:38:49.151937 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:49.151832 2577 projected.go:196] Error preparing data for projected volume kube-api-access-4qdq7 for pod openshift-network-diagnostics/network-check-target-4cbq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:49.151937 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:49.151896 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7 podName:f70226b0-5f4c-465c-9ec7-7ceee1bb376f nodeName:}" failed. No retries permitted until 2026-11-03 08:38:53.15187761 +0000 UTC m=+9.449774623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qdq7" (UniqueName: "kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7") pod "network-check-target-4cbq5" (UID: "f70226b0-5f4c-465c-9ec7-7ceee1bb376f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:49.235590 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:49.234952 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:49.235590 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:49.235128 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:38:49.235590 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:49.235226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:49.235590 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:49.235316 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:38:49.235590 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:49.235385 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:49.235590 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:49.235465 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:38:51.233978 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:51.233540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:51.233978 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:51.233670 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:38:51.233978 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:51.233724 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:51.233978 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:51.233793 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:38:51.233978 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:51.233864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:51.233978 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:51.233946 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:38:51.321562 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:51.321512 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b188283236722665eb092553b8ae03f" containerID="6ad903de0367c48d3dbc3e66dc042aaed6bc3ff78907c0020bafa27cab41410f" exitCode=0 Nov 03 08:38:51.321874 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:51.321832 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" event={"ID":"0b188283236722665eb092553b8ae03f","Type":"ContainerDied","Data":"6ad903de0367c48d3dbc3e66dc042aaed6bc3ff78907c0020bafa27cab41410f"} Nov 03 08:38:51.334652 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:51.334624 2577 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="31b9363fc26d3d999dbe30e48dcd58c9822c7dfa65a24b364b164cd6beff5fa3" exitCode=0 Nov 03 08:38:51.334729 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:51.334715 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4chsq" event={"ID":"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc","Type":"ContainerDied","Data":"31b9363fc26d3d999dbe30e48dcd58c9822c7dfa65a24b364b164cd6beff5fa3"} Nov 03 08:38:51.340687 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:51.340653 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hp76w" event={"ID":"1330da31-fcb6-4c7b-9db7-0d4d96262234","Type":"ContainerStarted","Data":"ba7994711a6ea89ecefb168a5e17e0ee7ed558fb46cb03fd391a7b7ae60f8986"} Nov 03 08:38:51.709382 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:51.707984 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hp76w" podStartSLOduration=1.452395917 podStartE2EDuration="6.707959425s" totalImagesPullingTime="5.255563508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:51.578682805 +0000 UTC m=+7.876579841" watchObservedRunningTime="2026-11-03 08:38:51.707959425 +0000 UTC m=+8.005856489" Nov 03 08:38:52.345790 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:52.345749 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" event={"ID":"6652f123a319d675474cd7afd732c542","Type":"ContainerStarted","Data":"d809d0321bdc230cb3eae37baceb04ebb96210124e7663c884c5e6e04954c023"} Nov 03 08:38:52.354876 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:52.354783 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wkb5w" event={"ID":"d3a0d3db-7b81-4e10-b240-b665c083bda2","Type":"ContainerStarted","Data":"760c72cde3f57cfaf2a226966846269a22064c59635c26f41dff37343589337a"} Nov 03 08:38:52.360185 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:52.360150 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" event={"ID":"0b188283236722665eb092553b8ae03f","Type":"ContainerStarted","Data":"770099cca10f3c04381449ea6b93fce72606da522fad12fe5ed1ae1316ddc62e"} Nov 03 08:38:52.472549 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:52.472089 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" podStartSLOduration=7.472066009 podStartE2EDuration="7.472066009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:52.470957326 +0000 UTC m=+8.768854360" watchObservedRunningTime="2026-11-03 08:38:52.472066009 +0000 UTC m=+8.769963048" Nov 03 08:38:52.602942 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:52.598644 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wkb5w" podStartSLOduration=1.294955242 podStartE2EDuration="7.598623169s" totalImagesPullingTime="6.303667927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:52.596902837 +0000 UTC m=+8.894799871" watchObservedRunningTime="2026-11-03 08:38:52.598623169 +0000 UTC m=+8.896520205" Nov 03 08:38:53.093902 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:53.093863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:53.094073 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:53.093932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:53.094073 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:53.094058 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:53.094203 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:53.094118 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret podName:25b42294-fcbd-4bc8-8652-9d4d7e762e3d nodeName:}" failed. No retries permitted until 2026-11-03 08:39:01.094099139 +0000 UTC m=+17.391996167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret") pod "global-pull-secret-syncer-c9fnb" (UID: "25b42294-fcbd-4bc8-8652-9d4d7e762e3d") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:53.094203 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:53.094167 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:53.094203 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:53.094189 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs podName:63810bfe-e55b-4524-bde4-8f072b4b63a3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:01.094182019 +0000 UTC m=+17.392079028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs") pod "network-metrics-daemon-q7vbp" (UID: "63810bfe-e55b-4524-bde4-8f072b4b63a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:53.194989 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:53.194943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdq7\" (UniqueName: \"kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7\") pod \"network-check-target-4cbq5\" (UID: \"f70226b0-5f4c-465c-9ec7-7ceee1bb376f\") " pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:53.195155 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:53.195132 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:38:53.195195 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:53.195157 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:38:53.195195 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:53.195172 2577 projected.go:196] Error preparing data for projected volume kube-api-access-4qdq7 for pod openshift-network-diagnostics/network-check-target-4cbq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:53.195253 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:53.195235 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7 podName:f70226b0-5f4c-465c-9ec7-7ceee1bb376f nodeName:}" failed. No retries permitted until 2026-11-03 08:39:01.195216093 +0000 UTC m=+17.493113124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qdq7" (UniqueName: "kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7") pod "network-check-target-4cbq5" (UID: "f70226b0-5f4c-465c-9ec7-7ceee1bb376f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:53.234324 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:53.234260 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:53.234493 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:53.234413 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:38:53.234599 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:53.234508 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:53.234644 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:53.234614 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:38:53.234716 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:53.234682 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:53.234779 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:53.234758 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:38:53.367917 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:53.366928 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" event={"ID":"bfbee40d-3926-4eae-babe-e598bfae8098","Type":"ContainerStarted","Data":"af4a3afc05001f7f3cdebbf6c614ec062b455beef846f3f77f47587cf57653bf"} Nov 03 08:38:53.963606 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:53.963569 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:53.964834 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:53.964808 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:54.094134 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:54.094061 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" podStartSLOduration=9.09404137 podStartE2EDuration="9.09404137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:52.725328769 +0000 UTC m=+9.023225812" watchObservedRunningTime="2026-11-03 08:38:54.09404137 +0000 UTC m=+10.391938404" Nov 03 08:38:54.372073 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:54.372038 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:54.372984 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:54.372752 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:38:55.233803 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:55.233766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:55.234000 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:55.233822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:55.234000 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:55.233766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:55.234000 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:55.233915 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:38:55.234148 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:55.234054 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:38:55.234185 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:55.234156 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:38:55.378253 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:55.377666 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8hwgr" event={"ID":"39df63cd-3b12-4930-8a23-04ed67d961a8","Type":"ContainerStarted","Data":"7ec1413455cf3ed7e70700cac9e3c9276f33956671757e14df0b29ffba4e625e"} Nov 03 08:38:55.434758 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:55.434596 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 08:38:56.202641 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:56.202278 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T08:38:55.434622453Z","UUID":"f984377d-f2f0-4a23-bc73-5f4f3fc124e9","Handler":null,"Name":"","Endpoint":""} Nov 03 08:38:56.205132 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:56.205095 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 08:38:56.205132 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:56.205126 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 08:38:56.382137 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:56.382055 2577 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="dcb483f3e40d8ec49fbc68a28a76f7f53601363125b0527b64ef78c43c07680e" exitCode=0 Nov 03 08:38:56.382660 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:56.382160 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4chsq" event={"ID":"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc","Type":"ContainerDied","Data":"dcb483f3e40d8ec49fbc68a28a76f7f53601363125b0527b64ef78c43c07680e"} Nov 03 08:38:56.389227 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:56.389190 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" event={"ID":"bfbee40d-3926-4eae-babe-e598bfae8098","Type":"ContainerStarted","Data":"41e08d7fbae6dff0f4fb0e1fd9ed6bec556a4baa1a09f53f4a194be0018e16df"} Nov 03 08:38:56.391629 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:56.391600 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2xrz9" event={"ID":"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c","Type":"ContainerStarted","Data":"23eaccc3e73e555817e55f8d99a682c7a97cd6e711adcefcd6fbd8d9f35b2b54"} Nov 03 08:38:56.524502 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:56.524446 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8hwgr" podStartSLOduration=2.098634309 podStartE2EDuration="11.524427574s" totalImagesPullingTime="9.425793265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:55.505775016 +0000 UTC m=+11.803672051" watchObservedRunningTime="2026-11-03 08:38:56.524427574 +0000 UTC m=+12.822324609" Nov 03 08:38:56.652491 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:56.652380 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2xrz9" podStartSLOduration=2.138161758 podStartE2EDuration="11.652369289s" totalImagesPullingTime="9.514207531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:56.651805209 +0000 UTC m=+12.949702239" watchObservedRunningTime="2026-11-03 08:38:56.652369289 +0000 UTC m=+12.950266321" Nov 03 08:38:57.233715 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:57.233681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:57.233888 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:57.233757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:57.233976 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:57.233880 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:57.233976 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:57.233892 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:38:57.234062 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:57.233975 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:38:57.234097 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:57.234075 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:38:57.395614 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:57.395576 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" event={"ID":"bfbee40d-3926-4eae-babe-e598bfae8098","Type":"ContainerStarted","Data":"9816c17b3270296e9d4550b9b12a9c41e51567c5be0f04c284db859bcd5c9a8a"} Nov 03 08:38:57.397399 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:57.397366 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zk7xq" event={"ID":"07f9e882-e987-44b0-a688-45e3c9abec5c","Type":"ContainerStarted","Data":"fc51a4b070ebb75503f4d63ea2222d7fca28c348fe8ec12bfb69459548a2292b"} Nov 03 08:38:57.514379 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:57.514284 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" podStartSLOduration=1.789186629 podStartE2EDuration="12.514270356s" totalImagesPullingTime="10.725083727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:57.514114834 +0000 UTC m=+13.812011864" watchObservedRunningTime="2026-11-03 08:38:57.514270356 +0000 UTC m=+13.812167386" Nov 03 08:38:57.630677 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:57.630628 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zk7xq" podStartSLOduration=3.3763998490000002 podStartE2EDuration="12.630585527s" totalImagesPullingTime="9.254185678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:57.630317775 +0000 UTC m=+13.928214832" watchObservedRunningTime="2026-11-03 08:38:57.630585527 +0000 UTC m=+13.928482579" Nov 03 08:38:58.406564 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:58.406168 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" event={"ID":"83c6ab8c-6c62-4163-a789-1edb75af2ad9","Type":"ContainerStarted","Data":"9b98a10c3e1dd0b280875d0bd7c24987c649ce03f846ee19ccdada46b875dfa9"} Nov 03 08:38:58.526417 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:58.526338 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" podStartSLOduration=1.465956099 podStartE2EDuration="13.526317822s" totalImagesPullingTime="12.060361723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:38:58.525788341 +0000 UTC m=+14.823685372" watchObservedRunningTime="2026-11-03 08:38:58.526317822 +0000 UTC m=+14.824214852" Nov 03 08:38:59.233901 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:59.233863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:38:59.233901 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:59.233880 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:38:59.234155 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:59.234012 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:38:59.234155 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:59.234048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:38:59.234155 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:59.234122 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:38:59.234285 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:38:59.234201 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:38:59.411557 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:59.411510 2577 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="f88ad47e1fc20a89b9a4693843555a1ea08513097824b3ddb15079a2e56cc7ec" exitCode=0 Nov 03 08:38:59.412030 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:38:59.411630 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4chsq" event={"ID":"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc","Type":"ContainerDied","Data":"f88ad47e1fc20a89b9a4693843555a1ea08513097824b3ddb15079a2e56cc7ec"} Nov 03 08:39:01.157574 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:01.157261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:01.157574 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:01.157533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:01.158048 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:01.157420 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:39:01.158048 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:01.157650 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs podName:63810bfe-e55b-4524-bde4-8f072b4b63a3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:17.157630368 +0000 UTC m=+33.455527378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs") pod "network-metrics-daemon-q7vbp" (UID: "63810bfe-e55b-4524-bde4-8f072b4b63a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:39:01.158048 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:01.157658 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:39:01.158048 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:01.157692 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret podName:25b42294-fcbd-4bc8-8652-9d4d7e762e3d nodeName:}" failed. No retries permitted until 2026-11-03 08:39:17.157682483 +0000 UTC m=+33.455579507 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret") pod "global-pull-secret-syncer-c9fnb" (UID: "25b42294-fcbd-4bc8-8652-9d4d7e762e3d") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:39:01.233308 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:01.233268 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:01.233522 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:01.233318 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:01.233522 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:01.233321 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:01.233522 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:01.233401 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:39:01.233669 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:01.233529 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:39:01.233669 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:01.233608 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:39:01.258619 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:01.258578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdq7\" (UniqueName: \"kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7\") pod \"network-check-target-4cbq5\" (UID: \"f70226b0-5f4c-465c-9ec7-7ceee1bb376f\") " pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:01.258805 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:01.258725 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:39:01.258805 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:01.258746 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:39:01.258805 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:01.258757 2577 projected.go:196] Error preparing data for projected volume kube-api-access-4qdq7 for pod openshift-network-diagnostics/network-check-target-4cbq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:39:01.258951 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:01.258813 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7 podName:f70226b0-5f4c-465c-9ec7-7ceee1bb376f nodeName:}" failed. No retries permitted until 2026-11-03 08:39:17.258794136 +0000 UTC m=+33.556691157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qdq7" (UniqueName: "kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7") pod "network-check-target-4cbq5" (UID: "f70226b0-5f4c-465c-9ec7-7ceee1bb376f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:39:01.419008 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:01.418916 2577 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="4f266ce650eacbe66d84c6ef92e19b97d43a88dffb646d70468b827bdd3f5aef" exitCode=0 Nov 03 08:39:01.419008 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:01.418970 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4chsq" event={"ID":"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc","Type":"ContainerDied","Data":"4f266ce650eacbe66d84c6ef92e19b97d43a88dffb646d70468b827bdd3f5aef"} Nov 03 08:39:03.233690 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:03.233662 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:03.234122 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:03.233710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:03.234122 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:03.233812 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:03.234122 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:03.233823 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:39:03.234122 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:03.233882 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:39:03.234122 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:03.233973 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:39:03.425541 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:03.425338 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" event={"ID":"1959beea-2daf-4e4a-a412-9bc2d35fc277","Type":"ContainerStarted","Data":"9924f8d786ca84c6b1fa480d284d59d35c7176a5481d1b54c22ed78cb9415ace"} Nov 03 08:39:03.425682 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:03.425582 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" event={"ID":"1959beea-2daf-4e4a-a412-9bc2d35fc277","Type":"ContainerStarted","Data":"20ea79fe424e18ed925157e50530d1d9142df867ae967bdb6ababdd241d98c3f"} Nov 03 08:39:03.425682 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:03.425613 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" event={"ID":"1959beea-2daf-4e4a-a412-9bc2d35fc277","Type":"ContainerStarted","Data":"955d348466830582c66b8a3b89b52e7425f0f77ecf2b4b4bd7c6c7ac5b55342d"} Nov 03 08:39:03.425682 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:03.425641 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" event={"ID":"1959beea-2daf-4e4a-a412-9bc2d35fc277","Type":"ContainerStarted","Data":"34b68d122b762d392db985494b294c6c4c200f893f8cbf81d4a6d5beb70e4070"} Nov 03 08:39:03.425682 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:03.425668 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" event={"ID":"1959beea-2daf-4e4a-a412-9bc2d35fc277","Type":"ContainerStarted","Data":"2c1e647049e06bb55a1ec36477f658fab17a3c338162c0775bf68d59f94f1004"} Nov 03 08:39:03.425806 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:03.425694 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" event={"ID":"1959beea-2daf-4e4a-a412-9bc2d35fc277","Type":"ContainerStarted","Data":"9f47bed4177d2cb74746751af9cc9e39a605c8d74393494aeba34c407522a8d2"} Nov 03 08:39:04.430021 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:04.429986 2577 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="cffe0ae561eab3368afbc538178e223e7a5b4aeda1d3a57fd240698031d8f54f" exitCode=0 Nov 03 08:39:04.430555 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:04.430035 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4chsq" event={"ID":"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc","Type":"ContainerDied","Data":"cffe0ae561eab3368afbc538178e223e7a5b4aeda1d3a57fd240698031d8f54f"} Nov 03 08:39:05.233154 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:05.233118 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:05.233327 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:05.233163 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:05.233327 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:05.233225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:05.233327 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:05.233245 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:39:05.233327 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:05.233291 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:39:05.233444 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:05.233380 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:39:05.435842 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:05.435811 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" event={"ID":"1959beea-2daf-4e4a-a412-9bc2d35fc277","Type":"ContainerStarted","Data":"71a86b87f9c10f2b3731e402f98233915771b12adceda926074182a612409701"} Nov 03 08:39:05.438866 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:05.438838 2577 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="30e2e6d64b3181c88fe61da946b3a38e5aa1bcf92c745e94e0b51b8bdb7b7422" exitCode=0 Nov 03 08:39:05.438986 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:05.438882 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4chsq" event={"ID":"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc","Type":"ContainerDied","Data":"30e2e6d64b3181c88fe61da946b3a38e5aa1bcf92c745e94e0b51b8bdb7b7422"} Nov 03 08:39:06.444066 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:06.444027 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4chsq" event={"ID":"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc","Type":"ContainerStarted","Data":"cd1aa03021f5afd10291b0e805e0a2a396efadcd21a8089a406c648934256ea5"} Nov 03 08:39:06.666254 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:06.666196 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4chsq" podStartSLOduration=7.853318431 podStartE2EDuration="21.666180304s" totalImagesPullingTime="13.812861873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:06.615732864 +0000 UTC m=+22.913629893" watchObservedRunningTime="2026-11-03 08:39:06.666180304 +0000 UTC m=+22.964077333" Nov 03 08:39:07.234059 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:07.234011 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:07.234273 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:07.234090 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:07.234273 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:07.234191 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:07.234273 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:07.234208 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:39:07.234273 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:07.234270 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:39:07.234470 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:07.234331 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:39:08.452218 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:08.452177 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" event={"ID":"1959beea-2daf-4e4a-a412-9bc2d35fc277","Type":"ContainerStarted","Data":"97f41ef58fabb4787297acfe413e486a375b6ace1edaf78469712bbbf45cebe9"} Nov 03 08:39:08.452837 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:08.452498 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:08.468899 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:08.468739 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:08.727317 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:08.727265 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" podStartSLOduration=6.572543411 podStartE2EDuration="23.727252009s" totalImagesPullingTime="17.154708598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:08.67687404 +0000 UTC m=+24.974771071" watchObservedRunningTime="2026-11-03 08:39:08.727252009 +0000 UTC m=+25.025149039" Nov 03 08:39:09.233663 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:09.233628 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:09.233850 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:09.233629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:09.233850 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:09.233746 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:39:09.233850 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:09.233833 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:39:09.234009 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:09.233629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:09.234009 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:09.233936 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:39:09.455293 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:09.455252 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:09.455720 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:09.455320 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:09.471497 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:09.471457 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:10.189092 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:10.189062 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c9fnb"] Nov 03 08:39:10.189251 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:10.189210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:10.189357 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:10.189326 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:39:10.191797 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:10.191770 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4cbq5"] Nov 03 08:39:10.191919 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:10.191870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:10.191982 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:10.191943 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:39:10.192394 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:10.192368 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q7vbp"] Nov 03 08:39:10.192507 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:10.192464 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:10.192589 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:10.192565 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:39:12.235430 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:12.235395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:12.235954 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:12.235639 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:39:12.235954 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:12.235405 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:12.235954 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:12.235762 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:39:12.235954 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:12.235395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:12.235954 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:12.235842 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:39:14.234577 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:14.234546 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:14.234971 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:14.234637 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:14.234971 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:14.234682 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:14.234971 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:14.234679 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c9fnb" podUID="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" Nov 03 08:39:14.234971 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:14.234783 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vbp" podUID="63810bfe-e55b-4524-bde4-8f072b4b63a3" Nov 03 08:39:14.234971 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:14.234878 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cbq5" podUID="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" Nov 03 08:39:15.493398 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.493376 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeReady" Nov 03 08:39:15.493856 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.493534 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 08:39:15.580820 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.580785 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-27vdv"] Nov 03 08:39:15.642523 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.642467 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5qm4n"] Nov 03 08:39:15.642660 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.642590 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-27vdv" Nov 03 08:39:15.673055 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.673005 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p5ncs"] Nov 03 08:39:15.673252 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.673123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5qm4n" Nov 03 08:39:15.683463 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.683423 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5"] Nov 03 08:39:15.683626 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.683602 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-p5ncs" Nov 03 08:39:15.697848 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.697822 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 08:39:15.697978 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.697904 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 08:39:15.703290 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.703266 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2"] Nov 03 08:39:15.703422 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.703334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:15.718014 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.717991 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt"] Nov 03 08:39:15.718136 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.718110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2" Nov 03 08:39:15.737950 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.737922 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-b2jvc"] Nov 03 08:39:15.738164 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.738096 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:15.751632 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.751609 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-b4ggb"] Nov 03 08:39:15.751745 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.751723 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.758072 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.758042 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 08:39:15.767705 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.767684 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtmjl\" (UniqueName: \"kubernetes.io/projected/248b4530-5359-4e26-8309-78d1a7128494-kube-api-access-vtmjl\") pod \"kas-connection-checker-685895c7c9-27vdv\" (UID: \"248b4530-5359-4e26-8309-78d1a7128494\") " pod="kube-system/kas-connection-checker-685895c7c9-27vdv" Nov 03 08:39:15.767794 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.767724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpm6\" (UniqueName: \"kubernetes.io/projected/61655665-1941-4f5f-bc6b-41e51b2b3774-kube-api-access-4mpm6\") pod \"kas-connection-checker-685895c7c9-5qm4n\" (UID: \"61655665-1941-4f5f-bc6b-41e51b2b3774\") " pod="kube-system/kas-connection-checker-685895c7c9-5qm4n" Nov 03 08:39:15.767828 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.767813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m764p\" (UniqueName: \"kubernetes.io/projected/c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a-kube-api-access-m764p\") pod \"kas-connection-checker-685895c7c9-p5ncs\" (UID: \"c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a\") " pod="kube-system/kas-connection-checker-685895c7c9-p5ncs" Nov 03 08:39:15.775855 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.775834 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r"] Nov 03 08:39:15.775978 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.775963 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.789990 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.789966 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx"] Nov 03 08:39:15.790093 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.790084 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:15.799049 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.799029 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:15.799125 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.799087 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 08:39:15.799178 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.799165 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:15.799212 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.799174 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:15.799287 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.799275 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 08:39:15.799507 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.799496 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 08:39:15.799547 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.799498 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:15.799756 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.799740 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 08:39:15.804159 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.804142 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-vrpjd"] Nov 03 08:39:15.804253 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.804238 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:15.805952 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.805922 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:15.806041 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.806025 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:15.806145 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.806132 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 08:39:15.826093 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.826072 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d6db59d9d-pk78q"] Nov 03 08:39:15.826203 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.826189 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:15.850053 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.850031 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 08:39:15.850244 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.850229 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64c88f488c-xq5g2"] Nov 03 08:39:15.850391 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.850374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:15.858818 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.858798 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 08:39:15.858924 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.858859 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 08:39:15.858975 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.858931 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 08:39:15.858975 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.858953 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g5h9g\"" Nov 03 08:39:15.858975 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.858956 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 08:39:15.859110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.858968 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vvc8d\"" Nov 03 08:39:15.859110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.858986 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tt7x7\"" Nov 03 08:39:15.859110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.859022 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-crw87\"" Nov 03 08:39:15.859110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.859040 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 08:39:15.859110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.859025 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-r6qnc\"" Nov 03 08:39:15.859110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.859084 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jw5dx\"" Nov 03 08:39:15.862572 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.862558 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6b64c9d49-xhzz8"] Nov 03 08:39:15.862673 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.862661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:15.868263 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmjl\" (UniqueName: \"kubernetes.io/projected/248b4530-5359-4e26-8309-78d1a7128494-kube-api-access-vtmjl\") pod \"kas-connection-checker-685895c7c9-27vdv\" (UID: \"248b4530-5359-4e26-8309-78d1a7128494\") " pod="kube-system/kas-connection-checker-685895c7c9-27vdv" Nov 03 08:39:15.868337 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrgh\" (UniqueName: \"kubernetes.io/projected/57491634-c53c-4571-854c-ece83d3f09a2-kube-api-access-jmrgh\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:15.868374 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:15.868374 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-tmp\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.868489 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a21ac76c-2765-417e-832a-32e140173e03-serving-cert\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.868527 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.868581 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpm6\" (UniqueName: \"kubernetes.io/projected/61655665-1941-4f5f-bc6b-41e51b2b3774-kube-api-access-4mpm6\") pod \"kas-connection-checker-685895c7c9-5qm4n\" (UID: \"61655665-1941-4f5f-bc6b-41e51b2b3774\") " pod="kube-system/kas-connection-checker-685895c7c9-5qm4n" Nov 03 08:39:15.868633 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46trz\" (UniqueName: \"kubernetes.io/projected/df61dbd1-26b9-4d1b-847c-785687d935ab-kube-api-access-46trz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:15.868665 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868630 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrtbr\" (UniqueName: \"kubernetes.io/projected/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-kube-api-access-mrtbr\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:15.868694 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868663 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbfl\" (UniqueName: \"kubernetes.io/projected/31ed67dd-1988-47af-a5a6-e00ee72236a3-kube-api-access-trbfl\") pod \"volume-data-source-validator-58c9f5f46d-b7vq2\" (UID: \"31ed67dd-1988-47af-a5a6-e00ee72236a3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2" Nov 03 08:39:15.868694 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-snapshots\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.868756 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df61dbd1-26b9-4d1b-847c-785687d935ab-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:15.868756 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-service-ca-bundle\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.868856 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-config\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.868856 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22edddd5-3b05-4942-910a-651d537867c8-serving-cert\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.868953 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868858 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qtzp\" (UniqueName: \"kubernetes.io/projected/22edddd5-3b05-4942-910a-651d537867c8-kube-api-access-4qtzp\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.868953 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:15.868953 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.868917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:15.869097 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.869009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m764p\" (UniqueName: \"kubernetes.io/projected/c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a-kube-api-access-m764p\") pod \"kas-connection-checker-685895c7c9-p5ncs\" (UID: \"c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a\") " pod="kube-system/kas-connection-checker-685895c7c9-p5ncs" Nov 03 08:39:15.869097 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.869038 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7zqh\" (UniqueName: \"kubernetes.io/projected/a21ac76c-2765-417e-832a-32e140173e03-kube-api-access-x7zqh\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.869097 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.869078 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df61dbd1-26b9-4d1b-847c-785687d935ab-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:15.869212 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.869103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-trusted-ca\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.869212 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.869128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57491634-c53c-4571-854c-ece83d3f09a2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:15.885004 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.884984 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg"] Nov 03 08:39:15.885112 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.885097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:15.899202 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.899181 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:15.899300 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.899278 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:15.899300 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.899285 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 08:39:15.899416 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.899322 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 08:39:15.899416 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.899345 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:15.899416 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.899358 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 08:39:15.899416 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.899370 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 08:39:15.899416 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.899406 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:15.899674 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.899433 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 08:39:15.899674 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.899570 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 08:39:15.900017 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.899999 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 08:39:15.904909 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.904895 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 08:39:15.909165 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.909143 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xs65g"] Nov 03 08:39:15.909274 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.909261 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg" Nov 03 08:39:15.923430 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.923411 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6rpvp"] Nov 03 08:39:15.923554 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.923541 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:15.929244 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.929220 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmjl\" (UniqueName: \"kubernetes.io/projected/248b4530-5359-4e26-8309-78d1a7128494-kube-api-access-vtmjl\") pod \"kas-connection-checker-685895c7c9-27vdv\" (UID: \"248b4530-5359-4e26-8309-78d1a7128494\") " pod="kube-system/kas-connection-checker-685895c7c9-27vdv" Nov 03 08:39:15.929244 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.929237 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpm6\" (UniqueName: \"kubernetes.io/projected/61655665-1941-4f5f-bc6b-41e51b2b3774-kube-api-access-4mpm6\") pod \"kas-connection-checker-685895c7c9-5qm4n\" (UID: \"61655665-1941-4f5f-bc6b-41e51b2b3774\") " pod="kube-system/kas-connection-checker-685895c7c9-5qm4n" Nov 03 08:39:15.929355 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.929251 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m764p\" (UniqueName: \"kubernetes.io/projected/c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a-kube-api-access-m764p\") pod \"kas-connection-checker-685895c7c9-p5ncs\" (UID: \"c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a\") " pod="kube-system/kas-connection-checker-685895c7c9-p5ncs" Nov 03 08:39:15.939657 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939640 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-27vdv"] Nov 03 08:39:15.939745 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939671 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5qm4n"] Nov 03 08:39:15.939745 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939697 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5"] Nov 03 08:39:15.939745 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939718 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2"] Nov 03 08:39:15.939745 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939739 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-vrpjd"] Nov 03 08:39:15.939861 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939758 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r"] Nov 03 08:39:15.939861 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939759 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:15.939861 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939779 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-b2jvc"] Nov 03 08:39:15.939861 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939799 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-b4ggb"] Nov 03 08:39:15.939861 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939817 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b64c9d49-xhzz8"] Nov 03 08:39:15.939861 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939837 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64c88f488c-xq5g2"] Nov 03 08:39:15.939861 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939857 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p5ncs"] Nov 03 08:39:15.940115 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939874 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d6db59d9d-pk78q"] Nov 03 08:39:15.940115 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939892 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt"] Nov 03 08:39:15.940115 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939918 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xs65g"] Nov 03 08:39:15.940115 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939942 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg"] Nov 03 08:39:15.940115 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939960 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx"] Nov 03 08:39:15.940115 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.939979 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6rpvp"] Nov 03 08:39:15.959885 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.959864 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 08:39:15.959885 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.959866 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 08:39:15.960052 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.959898 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rk9xz\"" Nov 03 08:39:15.960052 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.959900 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 08:39:15.960052 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.959920 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 08:39:15.960052 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.959927 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 08:39:15.960052 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.959947 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 08:39:15.960052 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.959951 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rdf8f\"" Nov 03 08:39:15.960052 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.959983 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hdxz9\"" Nov 03 08:39:15.960052 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.960021 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 08:39:15.969646 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.969626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrgh\" (UniqueName: \"kubernetes.io/projected/57491634-c53c-4571-854c-ece83d3f09a2-kube-api-access-jmrgh\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:15.969697 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.969653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:15.969697 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.969679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-installation-pull-secrets\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:15.969807 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.969793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z27j2\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-kube-api-access-z27j2\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:15.969840 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.969823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-tmp\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.969870 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.969841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.969870 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.969860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-registry-certificates\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:15.969978 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.969964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-stats-auth\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:15.970078 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46trz\" (UniqueName: \"kubernetes.io/projected/df61dbd1-26b9-4d1b-847c-785687d935ab-kube-api-access-46trz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:15.970108 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtbr\" (UniqueName: \"kubernetes.io/projected/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-kube-api-access-mrtbr\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:15.970140 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-certificates\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:15.970191 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970152 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/283e4bdb-5662-4b18-8775-7c86dd4323f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:15.970265 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtzp\" (UniqueName: \"kubernetes.io/projected/22edddd5-3b05-4942-910a-651d537867c8-kube-api-access-4qtzp\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.970377 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-image-registry-private-configuration\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:15.970377 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970283 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-tmp\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.970377 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbfl\" (UniqueName: \"kubernetes.io/projected/31ed67dd-1988-47af-a5a6-e00ee72236a3-kube-api-access-trbfl\") pod \"volume-data-source-validator-58c9f5f46d-b7vq2\" (UID: \"31ed67dd-1988-47af-a5a6-e00ee72236a3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2" Nov 03 08:39:15.970377 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-service-ca-bundle\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.970377 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:15.970626 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-config\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.970626 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-trusted-ca\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:15.970626 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:15.970626 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970574 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-bound-sa-token\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:15.970626 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7zqh\" (UniqueName: \"kubernetes.io/projected/a21ac76c-2765-417e-832a-32e140173e03-kube-api-access-x7zqh\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.970834 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-default-certificate\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:15.970834 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-trusted-ca\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.970834 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57491634-c53c-4571-854c-ece83d3f09a2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:15.970834 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0e56142-8762-447d-af25-d0ba1b09b530-ca-trust-extracted\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:15.970834 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970799 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-image-registry-private-configuration\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:15.970834 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970807 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.970834 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:15.971199 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970891 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a21ac76c-2765-417e-832a-32e140173e03-serving-cert\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.971199 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:15.970909 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 08:39:15.971199 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/16cb59f2-9d9f-4904-9fc8-bed9111deecd-config\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:15.971199 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.970951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:15.971199 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:15.970969 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls podName:7bebbc2e-c649-4f7f-bef3-0a8f158d6cae nodeName:}" failed. No retries permitted until 2026-11-03 08:39:16.470952827 +0000 UTC m=+32.768849852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgcwt" (UID: "7bebbc2e-c649-4f7f-bef3-0a8f158d6cae") : secret "samples-operator-tls" not found Nov 03 08:39:15.971199 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-snapshots\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.971199 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971006 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-service-ca-bundle\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.971199 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df61dbd1-26b9-4d1b-847c-785687d935ab-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:15.971199 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971110 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-config\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.971199 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16cb59f2-9d9f-4904-9fc8-bed9111deecd-serving-cert\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-bound-sa-token\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22edddd5-3b05-4942-910a-651d537867c8-serving-cert\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971297 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-installation-pull-secrets\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np8js\" (UniqueName: \"kubernetes.io/projected/16cb59f2-9d9f-4904-9fc8-bed9111deecd-kube-api-access-np8js\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971399 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74hn\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-kube-api-access-p74hn\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971456 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57491634-c53c-4571-854c-ece83d3f09a2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971469 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-trusted-ca\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-trusted-ca\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df61dbd1-26b9-4d1b-847c-785687d935ab-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5x8t\" (UniqueName: \"kubernetes.io/projected/70bfc17c-8e18-473a-a8fe-e386112cfaa4-kube-api-access-g5x8t\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971615 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-snapshots\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:15.971650 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:15.971700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-ca-trust-extracted\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:15.972190 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.971699 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df61dbd1-26b9-4d1b-847c-785687d935ab-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:15.972190 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:15.971873 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls podName:57491634-c53c-4571-854c-ece83d3f09a2 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:16.471857105 +0000 UTC m=+32.769754112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jrqm5" (UID: "57491634-c53c-4571-854c-ece83d3f09a2") : secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:15.972190 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.972036 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:15.973544 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.973518 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a21ac76c-2765-417e-832a-32e140173e03-serving-cert\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:15.973680 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.973665 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df61dbd1-26b9-4d1b-847c-785687d935ab-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:15.974012 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.973997 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22edddd5-3b05-4942-910a-651d537867c8-serving-cert\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:15.977617 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.977587 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2st7v\"" Nov 03 08:39:15.977617 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.977587 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 08:39:15.984725 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.984709 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-27vdv" Nov 03 08:39:15.999553 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.999513 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 08:39:15.999709 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.999692 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 08:39:15.999709 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.999702 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 08:39:15.999853 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.999725 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 08:39:15.999853 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.999725 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 08:39:15.999853 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.999749 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 08:39:15.999853 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.999734 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 08:39:15.999853 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:15.999773 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 08:39:16.011000 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.010984 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5qm4n" Nov 03 08:39:16.030298 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.030277 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-p5ncs" Nov 03 08:39:16.031791 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.031763 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrgh\" (UniqueName: \"kubernetes.io/projected/57491634-c53c-4571-854c-ece83d3f09a2-kube-api-access-jmrgh\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:16.032496 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.032454 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtzp\" (UniqueName: \"kubernetes.io/projected/22edddd5-3b05-4942-910a-651d537867c8-kube-api-access-4qtzp\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:16.032892 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.032876 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtbr\" (UniqueName: \"kubernetes.io/projected/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-kube-api-access-mrtbr\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:16.033217 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.033200 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46trz\" (UniqueName: \"kubernetes.io/projected/df61dbd1-26b9-4d1b-847c-785687d935ab-kube-api-access-46trz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:16.033379 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.033360 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbfl\" (UniqueName: \"kubernetes.io/projected/31ed67dd-1988-47af-a5a6-e00ee72236a3-kube-api-access-trbfl\") pod \"volume-data-source-validator-58c9f5f46d-b7vq2\" (UID: \"31ed67dd-1988-47af-a5a6-e00ee72236a3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2" Nov 03 08:39:16.033671 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.033655 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7zqh\" (UniqueName: \"kubernetes.io/projected/a21ac76c-2765-417e-832a-32e140173e03-kube-api-access-x7zqh\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:16.059552 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.059524 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 08:39:16.059863 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.059840 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sxt9c\"" Nov 03 08:39:16.066714 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.066680 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2" Nov 03 08:39:16.072979 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.072952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-bound-sa-token\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.073168 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.072987 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/587a6bc5-67cb-469b-b4b9-b35da4fb382b-config-volume\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.073168 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-default-certificate\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:16.073168 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0e56142-8762-447d-af25-d0ba1b09b530-ca-trust-extracted\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.073168 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-image-registry-private-configuration\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.073369 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpgs\" (UniqueName: \"kubernetes.io/projected/1bedc7c0-f94a-446e-8029-c3e82609f30f-kube-api-access-6xpgs\") pod \"ingress-canary-xs65g\" (UID: \"1bedc7c0-f94a-446e-8029-c3e82609f30f\") " pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:16.073369 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/16cb59f2-9d9f-4904-9fc8-bed9111deecd-config\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:16.073369 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073263 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.073369 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.073369 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16cb59f2-9d9f-4904-9fc8-bed9111deecd-serving-cert\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:16.073369 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlqkf\" (UniqueName: \"kubernetes.io/projected/587a6bc5-67cb-469b-b4b9-b35da4fb382b-kube-api-access-qlqkf\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.073676 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-bound-sa-token\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.073676 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-installation-pull-secrets\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.073676 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8js\" (UniqueName: \"kubernetes.io/projected/16cb59f2-9d9f-4904-9fc8-bed9111deecd-kube-api-access-np8js\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:16.073676 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/587a6bc5-67cb-469b-b4b9-b35da4fb382b-tmp-dir\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.073676 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073543 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0e56142-8762-447d-af25-d0ba1b09b530-ca-trust-extracted\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.073676 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:16.073676 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74hn\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-kube-api-access-p74hn\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.073676 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bedc7c0-f94a-446e-8029-c3e82609f30f-cert\") pod \"ingress-canary-xs65g\" (UID: \"1bedc7c0-f94a-446e-8029-c3e82609f30f\") " pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:16.073676 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-trusted-ca\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.074110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5x8t\" (UniqueName: \"kubernetes.io/projected/70bfc17c-8e18-473a-a8fe-e386112cfaa4-kube-api-access-g5x8t\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:16.074110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-ca-trust-extracted\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.074110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.074110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-installation-pull-secrets\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.074110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27j2\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-kube-api-access-z27j2\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.074110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073822 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpng7\" (UniqueName: \"kubernetes.io/projected/204918bd-5b87-4358-b80c-cd6262afb3b0-kube-api-access-tpng7\") pod \"network-check-source-6f78d59f99-kvwlg\" (UID: \"204918bd-5b87-4358-b80c-cd6262afb3b0\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg" Nov 03 08:39:16.074110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-registry-certificates\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.074110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.073884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-stats-auth\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:16.074594 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.074461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-certificates\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.074594 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.074510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/283e4bdb-5662-4b18-8775-7c86dd4323f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:16.074594 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.074536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-image-registry-private-configuration\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.074794 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.074591 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:16.074794 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.074164 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/16cb59f2-9d9f-4904-9fc8-bed9111deecd-config\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:16.074794 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.074626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-trusted-ca\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.074794 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.074656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:16.074794 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.074785 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 08:39:16.075318 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.074846 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert podName:283e4bdb-5662-4b18-8775-7c86dd4323f3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:16.574827145 +0000 UTC m=+32.872724167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vrpjd" (UID: "283e4bdb-5662-4b18-8775-7c86dd4323f3") : secret "networking-console-plugin-cert" not found Nov 03 08:39:16.078971 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.078742 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-installation-pull-secrets\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.081585 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.081497 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-certificates\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.083375 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.082569 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/283e4bdb-5662-4b18-8775-7c86dd4323f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:16.083375 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.082616 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:16.083375 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.082754 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6db59d9d-pk78q: secret "image-registry-tls" not found Nov 03 08:39:16.083375 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.082917 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls podName:a0e56142-8762-447d-af25-d0ba1b09b530 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:16.58279322 +0000 UTC m=+32.880690258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls") pod "image-registry-6d6db59d9d-pk78q" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530") : secret "image-registry-tls" not found Nov 03 08:39:16.083375 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.082954 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:16.083375 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.083046 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c88f488c-xq5g2: secret "image-registry-tls" not found Nov 03 08:39:16.083375 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.083104 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls podName:1b1845f9-4e7a-40b2-922a-95e1cb82eeea nodeName:}" failed. No retries permitted until 2026-11-03 08:39:16.583085303 +0000 UTC m=+32.880982314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls") pod "image-registry-64c88f488c-xq5g2" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea") : secret "image-registry-tls" not found Nov 03 08:39:16.086765 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.086741 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-installation-pull-secrets\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.086911 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.086869 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-default-certificate\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:16.091663 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.087972 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-trusted-ca\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.091663 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.088146 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 08:39:16.091663 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.088207 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:16.588188923 +0000 UTC m=+32.886085934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : secret "router-metrics-certs-default" not found Nov 03 08:39:16.091663 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.088736 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:16.588550521 +0000 UTC m=+32.886447593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : configmap references non-existent config key: service-ca.crt Nov 03 08:39:16.091663 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.089198 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-stats-auth\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:16.091663 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.089667 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-registry-certificates\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.091663 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.089693 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16cb59f2-9d9f-4904-9fc8-bed9111deecd-serving-cert\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:16.092386 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.092353 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-image-registry-private-configuration\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.092529 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.092507 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-ca-trust-extracted\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.093329 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.093305 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-trusted-ca\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.094952 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.094928 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-image-registry-private-configuration\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.121878 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.121836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:16.139168 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.139133 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-bound-sa-token\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.140700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.140673 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:16.142999 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.142937 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5x8t\" (UniqueName: \"kubernetes.io/projected/70bfc17c-8e18-473a-a8fe-e386112cfaa4-kube-api-access-g5x8t\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:16.144078 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.144048 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-bound-sa-token\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.144517 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.144496 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8js\" (UniqueName: \"kubernetes.io/projected/16cb59f2-9d9f-4904-9fc8-bed9111deecd-kube-api-access-np8js\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:16.149694 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.149649 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74hn\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-kube-api-access-p74hn\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.151063 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.151028 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27j2\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-kube-api-access-z27j2\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.161020 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.158769 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:16.175843 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.175814 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:16.176412 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.175988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/587a6bc5-67cb-469b-b4b9-b35da4fb382b-tmp-dir\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.176412 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.176041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bedc7c0-f94a-446e-8029-c3e82609f30f-cert\") pod \"ingress-canary-xs65g\" (UID: \"1bedc7c0-f94a-446e-8029-c3e82609f30f\") " pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:16.176412 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.176362 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/587a6bc5-67cb-469b-b4b9-b35da4fb382b-tmp-dir\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.176610 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.176536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpng7\" (UniqueName: \"kubernetes.io/projected/204918bd-5b87-4358-b80c-cd6262afb3b0-kube-api-access-tpng7\") pod \"network-check-source-6f78d59f99-kvwlg\" (UID: \"204918bd-5b87-4358-b80c-cd6262afb3b0\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg" Nov 03 08:39:16.176610 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.176607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/587a6bc5-67cb-469b-b4b9-b35da4fb382b-config-volume\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.176667 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.176640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpgs\" (UniqueName: \"kubernetes.io/projected/1bedc7c0-f94a-446e-8029-c3e82609f30f-kube-api-access-6xpgs\") pod \"ingress-canary-xs65g\" (UID: \"1bedc7c0-f94a-446e-8029-c3e82609f30f\") " pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:16.176700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.176675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.176700 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.176693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqkf\" (UniqueName: \"kubernetes.io/projected/587a6bc5-67cb-469b-b4b9-b35da4fb382b-kube-api-access-qlqkf\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.177545 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.177142 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/587a6bc5-67cb-469b-b4b9-b35da4fb382b-config-volume\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.177545 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.177305 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:16.177545 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.177351 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls podName:587a6bc5-67cb-469b-b4b9-b35da4fb382b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:16.677338832 +0000 UTC m=+32.975235862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls") pod "dns-default-6rpvp" (UID: "587a6bc5-67cb-469b-b4b9-b35da4fb382b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:16.180052 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.180034 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bedc7c0-f94a-446e-8029-c3e82609f30f-cert\") pod \"ingress-canary-xs65g\" (UID: \"1bedc7c0-f94a-446e-8029-c3e82609f30f\") " pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:16.233985 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.233718 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:16.234113 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.234019 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:16.234736 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.234347 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:16.237018 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.236988 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpgs\" (UniqueName: \"kubernetes.io/projected/1bedc7c0-f94a-446e-8029-c3e82609f30f-kube-api-access-6xpgs\") pod \"ingress-canary-xs65g\" (UID: \"1bedc7c0-f94a-446e-8029-c3e82609f30f\") " pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:16.237941 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.237918 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqkf\" (UniqueName: \"kubernetes.io/projected/587a6bc5-67cb-469b-b4b9-b35da4fb382b-kube-api-access-qlqkf\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.239055 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.239025 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpng7\" (UniqueName: \"kubernetes.io/projected/204918bd-5b87-4358-b80c-cd6262afb3b0-kube-api-access-tpng7\") pod \"network-check-source-6f78d59f99-kvwlg\" (UID: \"204918bd-5b87-4358-b80c-cd6262afb3b0\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg" Nov 03 08:39:16.276024 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.275872 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg" Nov 03 08:39:16.300266 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.294859 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-htlrf\"" Nov 03 08:39:16.300266 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.294929 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 08:39:16.300266 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.295011 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zgn57\"" Nov 03 08:39:16.300266 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.295111 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jv5wj\"" Nov 03 08:39:16.300266 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.295214 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 08:39:16.323534 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.314118 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:16.444881 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.444797 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-27vdv"] Nov 03 08:39:16.445767 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.445728 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5qm4n"] Nov 03 08:39:16.449518 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:16.449441 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248b4530_5359_4e26_8309_78d1a7128494.slice/crio-9e518abbca50250f10282cc24e25ffbecb12c50ad2979a9a55dc4a797555591e WatchSource:0}: Error finding container 9e518abbca50250f10282cc24e25ffbecb12c50ad2979a9a55dc4a797555591e: Status 404 returned error can't find the container with id 9e518abbca50250f10282cc24e25ffbecb12c50ad2979a9a55dc4a797555591e Nov 03 08:39:16.472015 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.471964 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-27vdv" event={"ID":"248b4530-5359-4e26-8309-78d1a7128494","Type":"ContainerStarted","Data":"9e518abbca50250f10282cc24e25ffbecb12c50ad2979a9a55dc4a797555591e"} Nov 03 08:39:16.473386 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.473360 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5qm4n" event={"ID":"61655665-1941-4f5f-bc6b-41e51b2b3774","Type":"ContainerStarted","Data":"be3e18905cfc85ed449d206ac748d373ad3e4d5e2e7ad12b44178c4c1657fc2d"} Nov 03 08:39:16.478337 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.478273 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p5ncs"] Nov 03 08:39:16.484211 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.483914 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:16.484211 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.483975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:16.484211 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.484094 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 08:39:16.484211 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.484100 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:16.484211 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.484153 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls podName:57491634-c53c-4571-854c-ece83d3f09a2 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:17.484140228 +0000 UTC m=+33.782037236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jrqm5" (UID: "57491634-c53c-4571-854c-ece83d3f09a2") : secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:16.484211 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.484168 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls podName:7bebbc2e-c649-4f7f-bef3-0a8f158d6cae nodeName:}" failed. No retries permitted until 2026-11-03 08:39:17.484159961 +0000 UTC m=+33.782056968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgcwt" (UID: "7bebbc2e-c649-4f7f-bef3-0a8f158d6cae") : secret "samples-operator-tls" not found Nov 03 08:39:16.486076 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:16.486056 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41e7f71_1eaa_4208_9fd8_dc6a2bee1e0a.slice/crio-087ab6565ab10ccc9d8fd9983e3728e0549df17a15e0e55e01d3201330779bcd WatchSource:0}: Error finding container 087ab6565ab10ccc9d8fd9983e3728e0549df17a15e0e55e01d3201330779bcd: Status 404 returned error can't find the container with id 087ab6565ab10ccc9d8fd9983e3728e0549df17a15e0e55e01d3201330779bcd Nov 03 08:39:16.490143 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.490122 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2"] Nov 03 08:39:16.496853 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:16.496824 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ed67dd_1988_47af_a5a6_e00ee72236a3.slice/crio-5eceb63331ed3385585589b32ec5638ad79823c60bf15484fc86898c82f60468 WatchSource:0}: Error finding container 5eceb63331ed3385585589b32ec5638ad79823c60bf15484fc86898c82f60468: Status 404 returned error can't find the container with id 5eceb63331ed3385585589b32ec5638ad79823c60bf15484fc86898c82f60468 Nov 03 08:39:16.540199 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.540172 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-b2jvc"] Nov 03 08:39:16.545641 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:16.545610 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21ac76c_2765_417e_832a_32e140173e03.slice/crio-0895bf54ec36b69a68bae88e8cdefe6315a189519b23cb005e695488bc9e2991 WatchSource:0}: Error finding container 0895bf54ec36b69a68bae88e8cdefe6315a189519b23cb005e695488bc9e2991: Status 404 returned error can't find the container with id 0895bf54ec36b69a68bae88e8cdefe6315a189519b23cb005e695488bc9e2991 Nov 03 08:39:16.580469 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.580413 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-b4ggb"] Nov 03 08:39:16.583389 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:16.583363 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22edddd5_3b05_4942_910a_651d537867c8.slice/crio-127b3ca309da266ab20064e258f8e4e83965a7722ba6ba0af7430a3178229f7b WatchSource:0}: Error finding container 127b3ca309da266ab20064e258f8e4e83965a7722ba6ba0af7430a3178229f7b: Status 404 returned error can't find the container with id 127b3ca309da266ab20064e258f8e4e83965a7722ba6ba0af7430a3178229f7b Nov 03 08:39:16.584448 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.584428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:16.584573 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.584534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:16.584631 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.584582 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:16.584631 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.584599 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c88f488c-xq5g2: secret "image-registry-tls" not found Nov 03 08:39:16.584745 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.584648 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls podName:1b1845f9-4e7a-40b2-922a-95e1cb82eeea nodeName:}" failed. No retries permitted until 2026-11-03 08:39:17.584633992 +0000 UTC m=+33.882531013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls") pod "image-registry-64c88f488c-xq5g2" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea") : secret "image-registry-tls" not found Nov 03 08:39:16.584745 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.584656 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:16.584745 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.584663 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 08:39:16.584745 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.584584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:16.584745 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.584669 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6db59d9d-pk78q: secret "image-registry-tls" not found Nov 03 08:39:16.584745 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.584716 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert podName:283e4bdb-5662-4b18-8775-7c86dd4323f3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:17.584701592 +0000 UTC m=+33.882598613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vrpjd" (UID: "283e4bdb-5662-4b18-8775-7c86dd4323f3") : secret "networking-console-plugin-cert" not found Nov 03 08:39:16.584745 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.584741 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls podName:a0e56142-8762-447d-af25-d0ba1b09b530 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:17.584728185 +0000 UTC m=+33.882625206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls") pod "image-registry-6d6db59d9d-pk78q" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530") : secret "image-registry-tls" not found Nov 03 08:39:16.619067 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.619044 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r"] Nov 03 08:39:16.623093 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:16.623068 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf61dbd1_26b9_4d1b_847c_785687d935ab.slice/crio-13f3d88234344e743fa45f800ae093325a8a8b7c5ada6bb5aa924aa31b9c986f WatchSource:0}: Error finding container 13f3d88234344e743fa45f800ae093325a8a8b7c5ada6bb5aa924aa31b9c986f: Status 404 returned error can't find the container with id 13f3d88234344e743fa45f800ae093325a8a8b7c5ada6bb5aa924aa31b9c986f Nov 03 08:39:16.679219 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.679196 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx"] Nov 03 08:39:16.681230 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:16.681198 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cb59f2_9d9f_4904_9fc8_bed9111deecd.slice/crio-97393e6231049f704e28e06e0cf4cce23b25376adc82f5f138587bb576d1a834 WatchSource:0}: Error finding container 97393e6231049f704e28e06e0cf4cce23b25376adc82f5f138587bb576d1a834: Status 404 returned error can't find the container with id 97393e6231049f704e28e06e0cf4cce23b25376adc82f5f138587bb576d1a834 Nov 03 08:39:16.686113 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.686093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:16.686196 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.686131 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:16.686196 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.686189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:16.686270 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.686245 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:16.686298 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.686282 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 08:39:16.686298 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.686297 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls podName:587a6bc5-67cb-469b-b4b9-b35da4fb382b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:17.686282592 +0000 UTC m=+33.984179603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls") pod "dns-default-6rpvp" (UID: "587a6bc5-67cb-469b-b4b9-b35da4fb382b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:16.686362 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.686310 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:17.68630448 +0000 UTC m=+33.984201493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : configmap references non-existent config key: service-ca.crt Nov 03 08:39:16.686362 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:16.686321 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:17.68631539 +0000 UTC m=+33.984212398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : secret "router-metrics-certs-default" not found Nov 03 08:39:16.696283 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.696269 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg"] Nov 03 08:39:16.698235 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:16.698213 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204918bd_5b87_4358_b80c_cd6262afb3b0.slice/crio-d2dc18521563827be82c995773f10820ae9dc806a948964a94cf095b6d925772 WatchSource:0}: Error finding container d2dc18521563827be82c995773f10820ae9dc806a948964a94cf095b6d925772: Status 404 returned error can't find the container with id d2dc18521563827be82c995773f10820ae9dc806a948964a94cf095b6d925772 Nov 03 08:39:16.727748 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:16.727725 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xs65g"] Nov 03 08:39:16.730107 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:16.730078 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bedc7c0_f94a_446e_8029_c3e82609f30f.slice/crio-ff30c44d7327fd48a3b59fc2e256027971bcbfca4a60111ac8e45586dee51bf9 WatchSource:0}: Error finding container ff30c44d7327fd48a3b59fc2e256027971bcbfca4a60111ac8e45586dee51bf9: Status 404 returned error can't find the container with id ff30c44d7327fd48a3b59fc2e256027971bcbfca4a60111ac8e45586dee51bf9 Nov 03 08:39:17.190091 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.190057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:17.190259 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.190156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:17.190259 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.190223 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 08:39:17.190327 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.190296 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs podName:63810bfe-e55b-4524-bde4-8f072b4b63a3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:49.190275486 +0000 UTC m=+65.488172512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs") pod "network-metrics-daemon-q7vbp" (UID: "63810bfe-e55b-4524-bde4-8f072b4b63a3") : secret "metrics-daemon-secret" not found Nov 03 08:39:17.192628 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.192596 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:17.291521 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.291466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdq7\" (UniqueName: \"kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7\") pod \"network-check-target-4cbq5\" (UID: \"f70226b0-5f4c-465c-9ec7-7ceee1bb376f\") " pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:17.294144 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.294111 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qdq7\" (UniqueName: \"kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7\") pod \"network-check-target-4cbq5\" (UID: \"f70226b0-5f4c-465c-9ec7-7ceee1bb376f\") " pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:17.468655 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.468562 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:17.477732 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.477693 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5qm4n" event={"ID":"61655665-1941-4f5f-bc6b-41e51b2b3774","Type":"ContainerStarted","Data":"4c250a23842c347ae115181496931b9dc74cebc3aef883e1fa318365705ceded"} Nov 03 08:39:17.479113 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.479077 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xs65g" event={"ID":"1bedc7c0-f94a-446e-8029-c3e82609f30f","Type":"ContainerStarted","Data":"ff30c44d7327fd48a3b59fc2e256027971bcbfca4a60111ac8e45586dee51bf9"} Nov 03 08:39:17.480497 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.480452 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg" event={"ID":"204918bd-5b87-4358-b80c-cd6262afb3b0","Type":"ContainerStarted","Data":"d2dc18521563827be82c995773f10820ae9dc806a948964a94cf095b6d925772"} Nov 03 08:39:17.481639 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.481616 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" event={"ID":"16cb59f2-9d9f-4904-9fc8-bed9111deecd","Type":"ContainerStarted","Data":"97393e6231049f704e28e06e0cf4cce23b25376adc82f5f138587bb576d1a834"} Nov 03 08:39:17.482808 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.482787 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" event={"ID":"df61dbd1-26b9-4d1b-847c-785687d935ab","Type":"ContainerStarted","Data":"13f3d88234344e743fa45f800ae093325a8a8b7c5ada6bb5aa924aa31b9c986f"} Nov 03 08:39:17.484008 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.483983 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-b4ggb" event={"ID":"22edddd5-3b05-4942-910a-651d537867c8","Type":"ContainerStarted","Data":"127b3ca309da266ab20064e258f8e4e83965a7722ba6ba0af7430a3178229f7b"} Nov 03 08:39:17.485383 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.485359 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2" event={"ID":"31ed67dd-1988-47af-a5a6-e00ee72236a3","Type":"ContainerStarted","Data":"5eceb63331ed3385585589b32ec5638ad79823c60bf15484fc86898c82f60468"} Nov 03 08:39:17.486975 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.486954 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-27vdv" event={"ID":"248b4530-5359-4e26-8309-78d1a7128494","Type":"ContainerStarted","Data":"89e684036d927b17bfc00cd21ed6ec90297cd1848c24e6a4bd3e28c0bb1bb22e"} Nov 03 08:39:17.488368 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.488345 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" event={"ID":"a21ac76c-2765-417e-832a-32e140173e03","Type":"ContainerStarted","Data":"0895bf54ec36b69a68bae88e8cdefe6315a189519b23cb005e695488bc9e2991"} Nov 03 08:39:17.490005 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.489933 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-p5ncs" event={"ID":"c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a","Type":"ContainerStarted","Data":"9cb66513344cc6ab73ea51d843eac532c482ebced7407f188152989dd12aa325"} Nov 03 08:39:17.490005 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.489974 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-p5ncs" event={"ID":"c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a","Type":"ContainerStarted","Data":"087ab6565ab10ccc9d8fd9983e3728e0549df17a15e0e55e01d3201330779bcd"} Nov 03 08:39:17.494780 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.494743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:17.494977 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.494830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:17.494977 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.494893 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 08:39:17.494977 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.494958 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:17.494977 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.494971 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls podName:7bebbc2e-c649-4f7f-bef3-0a8f158d6cae nodeName:}" failed. No retries permitted until 2026-11-03 08:39:19.494950282 +0000 UTC m=+35.792847304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgcwt" (UID: "7bebbc2e-c649-4f7f-bef3-0a8f158d6cae") : secret "samples-operator-tls" not found Nov 03 08:39:17.494977 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.495017 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls podName:57491634-c53c-4571-854c-ece83d3f09a2 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:19.495001517 +0000 UTC m=+35.792898539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jrqm5" (UID: "57491634-c53c-4571-854c-ece83d3f09a2") : secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:17.523038 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.522998 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:17.596869 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.595994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:17.596869 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.596105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:17.596869 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.596173 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:17.599925 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.597925 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:17.599925 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.597991 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c88f488c-xq5g2: secret "image-registry-tls" not found Nov 03 08:39:17.599925 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.598170 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls podName:1b1845f9-4e7a-40b2-922a-95e1cb82eeea nodeName:}" failed. No retries permitted until 2026-11-03 08:39:19.598148368 +0000 UTC m=+35.896045389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls") pod "image-registry-64c88f488c-xq5g2" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea") : secret "image-registry-tls" not found Nov 03 08:39:17.599925 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.598416 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 08:39:17.599925 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.598462 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert podName:283e4bdb-5662-4b18-8775-7c86dd4323f3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:19.59844885 +0000 UTC m=+35.896345873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vrpjd" (UID: "283e4bdb-5662-4b18-8775-7c86dd4323f3") : secret "networking-console-plugin-cert" not found Nov 03 08:39:17.599925 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.598540 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:17.599925 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.598552 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6db59d9d-pk78q: secret "image-registry-tls" not found Nov 03 08:39:17.599925 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.598586 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls podName:a0e56142-8762-447d-af25-d0ba1b09b530 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:19.598572941 +0000 UTC m=+35.896469972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls") pod "image-registry-6d6db59d9d-pk78q" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530") : secret "image-registry-tls" not found Nov 03 08:39:17.699042 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.699003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:17.699231 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.699113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:17.699231 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.699161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:17.699345 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.699337 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:19.699318647 +0000 UTC m=+35.997215672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : configmap references non-existent config key: service-ca.crt Nov 03 08:39:17.699464 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.699419 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 08:39:17.699464 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.699452 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:19.699442086 +0000 UTC m=+35.997339107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : secret "router-metrics-certs-default" not found Nov 03 08:39:17.699591 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.699525 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:17.699591 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:17.699555 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls podName:587a6bc5-67cb-469b-b4b9-b35da4fb382b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:19.699545593 +0000 UTC m=+35.997442616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls") pod "dns-default-6rpvp" (UID: "587a6bc5-67cb-469b-b4b9-b35da4fb382b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:17.861939 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.861870 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5qm4n" podStartSLOduration=399.861846593 podStartE2EDuration="6m39.861846593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:17.817573662 +0000 UTC m=+34.115470691" watchObservedRunningTime="2026-11-03 08:39:17.861846593 +0000 UTC m=+34.159743624" Nov 03 08:39:17.863034 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.863006 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c9fnb"] Nov 03 08:39:17.877824 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:17.877790 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b42294_fcbd_4bc8_8652_9d4d7e762e3d.slice/crio-016182c16a32739ec7ea9747965e720524005017c05c32a0792ada01d1ad2953 WatchSource:0}: Error finding container 016182c16a32739ec7ea9747965e720524005017c05c32a0792ada01d1ad2953: Status 404 returned error can't find the container with id 016182c16a32739ec7ea9747965e720524005017c05c32a0792ada01d1ad2953 Nov 03 08:39:17.919496 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.919362 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4cbq5"] Nov 03 08:39:17.923579 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:17.923504 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70226b0_5f4c_465c_9ec7_7ceee1bb376f.slice/crio-652da50b44684bce010ffe38bdba42f1d68edce3a789c682cfe2608d44ae91eb WatchSource:0}: Error finding container 652da50b44684bce010ffe38bdba42f1d68edce3a789c682cfe2608d44ae91eb: Status 404 returned error can't find the container with id 652da50b44684bce010ffe38bdba42f1d68edce3a789c682cfe2608d44ae91eb Nov 03 08:39:17.936150 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:17.935715 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-p5ncs" podStartSLOduration=399.935696367 podStartE2EDuration="6m39.935696367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:17.934523507 +0000 UTC m=+34.232420538" watchObservedRunningTime="2026-11-03 08:39:17.935696367 +0000 UTC m=+34.233593399" Nov 03 08:39:18.051553 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:18.051501 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-27vdv" podStartSLOduration=400.051466103 podStartE2EDuration="6m40.051466103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:18.050974379 +0000 UTC m=+34.348871407" watchObservedRunningTime="2026-11-03 08:39:18.051466103 +0000 UTC m=+34.349363133" Nov 03 08:39:18.498812 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:18.498772 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4cbq5" event={"ID":"f70226b0-5f4c-465c-9ec7-7ceee1bb376f","Type":"ContainerStarted","Data":"652da50b44684bce010ffe38bdba42f1d68edce3a789c682cfe2608d44ae91eb"} Nov 03 08:39:18.502148 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:18.502115 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c9fnb" event={"ID":"25b42294-fcbd-4bc8-8652-9d4d7e762e3d","Type":"ContainerStarted","Data":"016182c16a32739ec7ea9747965e720524005017c05c32a0792ada01d1ad2953"} Nov 03 08:39:19.514897 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:19.514587 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2" event={"ID":"31ed67dd-1988-47af-a5a6-e00ee72236a3","Type":"ContainerStarted","Data":"04c84a154fde315d2ebb8469ef8aae3e031406d38a9fbee8b86e071f7d08ef1c"} Nov 03 08:39:19.522895 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:19.522865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:19.523051 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:19.522952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:19.523184 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.523167 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:19.523243 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.523233 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls podName:57491634-c53c-4571-854c-ece83d3f09a2 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:23.523213917 +0000 UTC m=+39.821110942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jrqm5" (UID: "57491634-c53c-4571-854c-ece83d3f09a2") : secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:19.523655 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.523635 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 08:39:19.523751 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.523694 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls podName:7bebbc2e-c649-4f7f-bef3-0a8f158d6cae nodeName:}" failed. No retries permitted until 2026-11-03 08:39:23.52367735 +0000 UTC m=+39.821574373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgcwt" (UID: "7bebbc2e-c649-4f7f-bef3-0a8f158d6cae") : secret "samples-operator-tls" not found Nov 03 08:39:19.623745 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:19.623716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:19.624006 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:19.623841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:19.624006 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.623900 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:19.624006 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.623924 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6db59d9d-pk78q: secret "image-registry-tls" not found Nov 03 08:39:19.624006 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.623996 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls podName:a0e56142-8762-447d-af25-d0ba1b09b530 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:23.623976291 +0000 UTC m=+39.921873317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls") pod "image-registry-6d6db59d9d-pk78q" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530") : secret "image-registry-tls" not found Nov 03 08:39:19.624460 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:19.624043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:19.624460 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.624235 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:19.624460 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.624250 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c88f488c-xq5g2: secret "image-registry-tls" not found Nov 03 08:39:19.624460 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.624339 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls podName:1b1845f9-4e7a-40b2-922a-95e1cb82eeea nodeName:}" failed. No retries permitted until 2026-11-03 08:39:23.624323675 +0000 UTC m=+39.922220715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls") pod "image-registry-64c88f488c-xq5g2" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea") : secret "image-registry-tls" not found Nov 03 08:39:19.624460 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.624394 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 08:39:19.624460 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.624442 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert podName:283e4bdb-5662-4b18-8775-7c86dd4323f3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:23.624430422 +0000 UTC m=+39.922327447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vrpjd" (UID: "283e4bdb-5662-4b18-8775-7c86dd4323f3") : secret "networking-console-plugin-cert" not found Nov 03 08:39:19.725304 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:19.725264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:19.725491 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:19.725375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:19.725491 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:19.725423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:19.725620 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.725610 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:23.72559014 +0000 UTC m=+40.023487165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : configmap references non-existent config key: service-ca.crt Nov 03 08:39:19.725723 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.725698 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 08:39:19.725811 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.725741 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:23.725729029 +0000 UTC m=+40.023626051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : secret "router-metrics-certs-default" not found Nov 03 08:39:19.725811 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.725798 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:19.725915 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:19.725828 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls podName:587a6bc5-67cb-469b-b4b9-b35da4fb382b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:23.725816958 +0000 UTC m=+40.023713983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls") pod "dns-default-6rpvp" (UID: "587a6bc5-67cb-469b-b4b9-b35da4fb382b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:21.526123 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:21.525896 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" event={"ID":"a21ac76c-2765-417e-832a-32e140173e03","Type":"ContainerStarted","Data":"a124c3eb36bb9864d21442d68b52a2da6e3cd6060bc72c25f020c01931d70ee5"} Nov 03 08:39:21.526123 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:21.525960 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:21.527663 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:21.527528 2577 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-b2jvc container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Nov 03 08:39:21.527663 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:21.527592 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podUID="a21ac76c-2765-417e-832a-32e140173e03" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Nov 03 08:39:21.644334 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:21.644263 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2" podStartSLOduration=372.161933815 podStartE2EDuration="6m14.644244102s" totalImagesPullingTime="2.482310287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:19.631807413 +0000 UTC m=+35.929704442" watchObservedRunningTime="2026-11-03 08:39:21.644244102 +0000 UTC m=+37.942141133" Nov 03 08:39:21.644925 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:21.644878 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podStartSLOduration=368.795413781 podStartE2EDuration="6m13.644865564s" totalImagesPullingTime="4.849451783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:21.643923281 +0000 UTC m=+37.941820309" watchObservedRunningTime="2026-11-03 08:39:21.644865564 +0000 UTC m=+37.942762597" Nov 03 08:39:22.534685 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:22.533400 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/0.log" Nov 03 08:39:22.534685 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:22.533620 2577 generic.go:356] "Generic (PLEG): container finished" podID="a21ac76c-2765-417e-832a-32e140173e03" containerID="a124c3eb36bb9864d21442d68b52a2da6e3cd6060bc72c25f020c01931d70ee5" exitCode=255 Nov 03 08:39:22.534685 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:22.533667 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" event={"ID":"a21ac76c-2765-417e-832a-32e140173e03","Type":"ContainerDied","Data":"a124c3eb36bb9864d21442d68b52a2da6e3cd6060bc72c25f020c01931d70ee5"} Nov 03 08:39:22.534685 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:22.533892 2577 scope.go:122] "RemoveContainer" containerID="a124c3eb36bb9864d21442d68b52a2da6e3cd6060bc72c25f020c01931d70ee5" Nov 03 08:39:23.539962 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.539930 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" event={"ID":"16cb59f2-9d9f-4904-9fc8-bed9111deecd","Type":"ContainerStarted","Data":"b5bdf845da4aca9cac4741b383cbb241817975e2d32455659b41e5d5ee1614b8"} Nov 03 08:39:23.544056 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.543603 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" event={"ID":"df61dbd1-26b9-4d1b-847c-785687d935ab","Type":"ContainerStarted","Data":"777a6be2956e0afa3006eb9183f62eb0a34872a9553d5dce2c1808daa768fcb1"} Nov 03 08:39:23.546042 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.545977 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-b4ggb" event={"ID":"22edddd5-3b05-4942-910a-651d537867c8","Type":"ContainerStarted","Data":"a401dae88d6eb1871e59e25194d85f40ab0ce4e0864f2441649738ebf38b5804"} Nov 03 08:39:23.548492 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.548452 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/1.log" Nov 03 08:39:23.549045 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.549007 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/0.log" Nov 03 08:39:23.549045 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.549035 2577 generic.go:356] "Generic (PLEG): container finished" podID="a21ac76c-2765-417e-832a-32e140173e03" containerID="98480b8ec484ff22c4640ad44514431fb0b225906819ec8a69dfdc4b153fff24" exitCode=255 Nov 03 08:39:23.549192 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.549077 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" event={"ID":"a21ac76c-2765-417e-832a-32e140173e03","Type":"ContainerDied","Data":"98480b8ec484ff22c4640ad44514431fb0b225906819ec8a69dfdc4b153fff24"} Nov 03 08:39:23.549192 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.549118 2577 scope.go:122] "RemoveContainer" containerID="a124c3eb36bb9864d21442d68b52a2da6e3cd6060bc72c25f020c01931d70ee5" Nov 03 08:39:23.549456 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.549431 2577 scope.go:122] "RemoveContainer" containerID="98480b8ec484ff22c4640ad44514431fb0b225906819ec8a69dfdc4b153fff24" Nov 03 08:39:23.549652 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.549623 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b2jvc_openshift-console-operator(a21ac76c-2765-417e-832a-32e140173e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podUID="a21ac76c-2765-417e-832a-32e140173e03" Nov 03 08:39:23.571095 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.571053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:23.571273 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.571125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:23.571327 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.571316 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:23.571380 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.571368 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls podName:57491634-c53c-4571-854c-ece83d3f09a2 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:31.571355705 +0000 UTC m=+47.869252726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jrqm5" (UID: "57491634-c53c-4571-854c-ece83d3f09a2") : secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:23.571437 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.571419 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 08:39:23.571586 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.571447 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls podName:7bebbc2e-c649-4f7f-bef3-0a8f158d6cae nodeName:}" failed. No retries permitted until 2026-11-03 08:39:31.571440322 +0000 UTC m=+47.869337343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgcwt" (UID: "7bebbc2e-c649-4f7f-bef3-0a8f158d6cae") : secret "samples-operator-tls" not found Nov 03 08:39:23.658308 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.657331 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" podStartSLOduration=369.422028399 podStartE2EDuration="6m15.657315223s" totalImagesPullingTime="6.235286824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:23.656770435 +0000 UTC m=+39.954667466" watchObservedRunningTime="2026-11-03 08:39:23.657315223 +0000 UTC m=+39.955212299" Nov 03 08:39:23.671946 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.671910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:23.672078 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.672050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:23.673220 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.673197 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:23.673220 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.673222 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6db59d9d-pk78q: secret "image-registry-tls" not found Nov 03 08:39:23.673385 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.673253 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 08:39:23.673385 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.673286 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls podName:a0e56142-8762-447d-af25-d0ba1b09b530 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:31.673265717 +0000 UTC m=+47.971162741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls") pod "image-registry-6d6db59d9d-pk78q" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530") : secret "image-registry-tls" not found Nov 03 08:39:23.673385 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.673313 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert podName:283e4bdb-5662-4b18-8775-7c86dd4323f3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:31.673293494 +0000 UTC m=+47.971190528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vrpjd" (UID: "283e4bdb-5662-4b18-8775-7c86dd4323f3") : secret "networking-console-plugin-cert" not found Nov 03 08:39:23.673581 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.673549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:23.674856 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.674839 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:23.674951 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.674859 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c88f488c-xq5g2: secret "image-registry-tls" not found Nov 03 08:39:23.674951 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.674901 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls podName:1b1845f9-4e7a-40b2-922a-95e1cb82eeea nodeName:}" failed. No retries permitted until 2026-11-03 08:39:31.674886432 +0000 UTC m=+47.972783467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls") pod "image-registry-64c88f488c-xq5g2" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea") : secret "image-registry-tls" not found Nov 03 08:39:23.775253 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.775213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:23.775439 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.775280 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:23.775439 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.775348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:23.775593 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.775499 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 08:39:23.775593 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.775551 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:31.775539123 +0000 UTC m=+48.073436144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : secret "router-metrics-certs-default" not found Nov 03 08:39:23.775696 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.775618 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:23.775696 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.775651 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls podName:587a6bc5-67cb-469b-b4b9-b35da4fb382b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:31.775638945 +0000 UTC m=+48.073535970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls") pod "dns-default-6rpvp" (UID: "587a6bc5-67cb-469b-b4b9-b35da4fb382b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:23.775795 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:23.775714 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:31.775707447 +0000 UTC m=+48.073604468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : configmap references non-existent config key: service-ca.crt Nov 03 08:39:23.776863 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:23.776562 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-b4ggb" podStartSLOduration=370.88444001 podStartE2EDuration="6m16.776545435s" totalImagesPullingTime="5.892105425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:23.77596006 +0000 UTC m=+40.073857091" watchObservedRunningTime="2026-11-03 08:39:23.776545435 +0000 UTC m=+40.074442640" Nov 03 08:39:24.018848 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.018790 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" podStartSLOduration=370.176621895 podStartE2EDuration="6m16.018773881s" totalImagesPullingTime="5.842151986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:24.017857272 +0000 UTC m=+40.315754302" watchObservedRunningTime="2026-11-03 08:39:24.018773881 +0000 UTC m=+40.316670951" Nov 03 08:39:24.022284 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.022257 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pc6m2"] Nov 03 08:39:24.043179 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.043148 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.089811 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.089749 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pc6m2"] Nov 03 08:39:24.099946 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.099912 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 08:39:24.100110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.100019 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lngsh\"" Nov 03 08:39:24.103501 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.103460 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 08:39:24.181119 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.181077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.181266 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.181220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm8nz\" (UniqueName: \"kubernetes.io/projected/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-api-access-bm8nz\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.181336 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.181294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19c1fbeb-055b-4c85-aebe-d4856252042a-data-volume\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.181413 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.181394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19c1fbeb-055b-4c85-aebe-d4856252042a-crio-socket\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.181511 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.181467 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.252009 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.251982 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-llp7m-mp4tf"] Nov 03 08:39:24.260422 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.260392 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.282927 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.282898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.283110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.282943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.283110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.283001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8nz\" (UniqueName: \"kubernetes.io/projected/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-api-access-bm8nz\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.283110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.283049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19c1fbeb-055b-4c85-aebe-d4856252042a-data-volume\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.283110 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.283093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19c1fbeb-055b-4c85-aebe-d4856252042a-crio-socket\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.283321 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.283256 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19c1fbeb-055b-4c85-aebe-d4856252042a-crio-socket\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.283372 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:24.283341 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 08:39:24.283419 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:24.283384 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls podName:19c1fbeb-055b-4c85-aebe-d4856252042a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:24.783370628 +0000 UTC m=+41.081267649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pc6m2" (UID: "19c1fbeb-055b-4c85-aebe-d4856252042a") : secret "insights-runtime-extractor-tls" not found Nov 03 08:39:24.283653 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:24.283627 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 08:39:24.283730 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:24.283670 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm podName:19c1fbeb-055b-4c85-aebe-d4856252042a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:24.783658449 +0000 UTC m=+41.081555470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pc6m2" (UID: "19c1fbeb-055b-4c85-aebe-d4856252042a") : configmap "kube-rbac-proxy" not found Nov 03 08:39:24.284013 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.283990 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19c1fbeb-055b-4c85-aebe-d4856252042a-data-volume\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.318027 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.317999 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-llp7m-mp4tf"] Nov 03 08:39:24.328603 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.328572 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7"] Nov 03 08:39:24.356508 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.354210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8nz\" (UniqueName: \"kubernetes.io/projected/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-api-access-bm8nz\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.356508 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.355592 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h"] Nov 03 08:39:24.356508 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.355731 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7" Nov 03 08:39:24.376177 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.376153 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h" Nov 03 08:39:24.384776 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.383736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-service-ca-bundle\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.384776 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.383799 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdt5w\" (UniqueName: \"kubernetes.io/projected/e0106338-5b18-4bbf-abfe-f2eee3213c8c-kube-api-access-wdt5w\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.384776 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.384017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0106338-5b18-4bbf-abfe-f2eee3213c8c-archives-path\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.384776 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.384089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-trusted-ca-bundle\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.401989 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.401937 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7"] Nov 03 08:39:24.411409 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.411329 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h"] Nov 03 08:39:24.411635 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.411615 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gc7sr\"" Nov 03 08:39:24.429706 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.429673 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:24.429706 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.429672 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 08:39:24.485266 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.485230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0106338-5b18-4bbf-abfe-f2eee3213c8c-archives-path\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.485507 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.485307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7tj\" (UniqueName: \"kubernetes.io/projected/3a650ee8-e4e1-4c94-a7da-836ea74de483-kube-api-access-ll7tj\") pod \"migrator-66df5d496b-ghcc7\" (UID: \"3a650ee8-e4e1-4c94-a7da-836ea74de483\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7" Nov 03 08:39:24.485507 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.485351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-trusted-ca-bundle\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.485507 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.485379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f27l\" (UniqueName: \"kubernetes.io/projected/bfcdc2e6-1f23-4da9-a478-0ad2175c5def-kube-api-access-6f27l\") pod \"migrator-66df5d496b-ldh5h\" (UID: \"bfcdc2e6-1f23-4da9-a478-0ad2175c5def\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h" Nov 03 08:39:24.485698 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.485573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-service-ca-bundle\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.485698 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.485626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdt5w\" (UniqueName: \"kubernetes.io/projected/e0106338-5b18-4bbf-abfe-f2eee3213c8c-kube-api-access-wdt5w\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.485795 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.485718 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0106338-5b18-4bbf-abfe-f2eee3213c8c-archives-path\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.486233 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.486187 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-trusted-ca-bundle\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.486344 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.486238 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-service-ca-bundle\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.546916 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.546841 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdt5w\" (UniqueName: \"kubernetes.io/projected/e0106338-5b18-4bbf-abfe-f2eee3213c8c-kube-api-access-wdt5w\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.554579 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.554558 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/1.log" Nov 03 08:39:24.555166 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.554929 2577 scope.go:122] "RemoveContainer" containerID="98480b8ec484ff22c4640ad44514431fb0b225906819ec8a69dfdc4b153fff24" Nov 03 08:39:24.555166 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:24.555106 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b2jvc_openshift-console-operator(a21ac76c-2765-417e-832a-32e140173e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podUID="a21ac76c-2765-417e-832a-32e140173e03" Nov 03 08:39:24.556892 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.556865 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xs65g" event={"ID":"1bedc7c0-f94a-446e-8029-c3e82609f30f","Type":"ContainerStarted","Data":"1dac1442860813ee771b7bb7a9168a7670f826e3d52e8ca4ac4feefec7e8e574"} Nov 03 08:39:24.559215 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.558877 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg" event={"ID":"204918bd-5b87-4358-b80c-cd6262afb3b0","Type":"ContainerStarted","Data":"92eab9dcf329cba9ad5f02047bb4dbdc14321170496ab52b91631e731d7a3fff"} Nov 03 08:39:24.586441 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.586401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7tj\" (UniqueName: \"kubernetes.io/projected/3a650ee8-e4e1-4c94-a7da-836ea74de483-kube-api-access-ll7tj\") pod \"migrator-66df5d496b-ghcc7\" (UID: \"3a650ee8-e4e1-4c94-a7da-836ea74de483\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7" Nov 03 08:39:24.586571 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.586449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f27l\" (UniqueName: \"kubernetes.io/projected/bfcdc2e6-1f23-4da9-a478-0ad2175c5def-kube-api-access-6f27l\") pod \"migrator-66df5d496b-ldh5h\" (UID: \"bfcdc2e6-1f23-4da9-a478-0ad2175c5def\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h" Nov 03 08:39:24.601401 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.601378 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:24.648519 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.646307 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f27l\" (UniqueName: \"kubernetes.io/projected/bfcdc2e6-1f23-4da9-a478-0ad2175c5def-kube-api-access-6f27l\") pod \"migrator-66df5d496b-ldh5h\" (UID: \"bfcdc2e6-1f23-4da9-a478-0ad2175c5def\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h" Nov 03 08:39:24.648519 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.646358 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7tj\" (UniqueName: \"kubernetes.io/projected/3a650ee8-e4e1-4c94-a7da-836ea74de483-kube-api-access-ll7tj\") pod \"migrator-66df5d496b-ghcc7\" (UID: \"3a650ee8-e4e1-4c94-a7da-836ea74de483\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7" Nov 03 08:39:24.685039 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.685014 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7" Nov 03 08:39:24.709760 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.709711 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h" Nov 03 08:39:24.789991 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.789946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.790174 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.790013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:24.790381 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:24.790245 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 08:39:24.790381 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:24.790320 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm podName:19c1fbeb-055b-4c85-aebe-d4856252042a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:25.790301178 +0000 UTC m=+42.088198204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pc6m2" (UID: "19c1fbeb-055b-4c85-aebe-d4856252042a") : configmap "kube-rbac-proxy" not found Nov 03 08:39:24.790381 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:24.790318 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 08:39:24.790381 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:24.790365 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls podName:19c1fbeb-055b-4c85-aebe-d4856252042a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:25.790353109 +0000 UTC m=+42.088250131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pc6m2" (UID: "19c1fbeb-055b-4c85-aebe-d4856252042a") : secret "insights-runtime-extractor-tls" not found Nov 03 08:39:24.855827 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.855755 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xs65g" podStartSLOduration=2.936502708 podStartE2EDuration="9.855737715s" totalImagesPullingTime="6.919235007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:24.855637858 +0000 UTC m=+41.153534891" watchObservedRunningTime="2026-11-03 08:39:24.855737715 +0000 UTC m=+41.153634746" Nov 03 08:39:24.971098 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:24.971037 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg" podStartSLOduration=367.367552213 podStartE2EDuration="6m14.971021763s" totalImagesPullingTime="7.60346955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:24.970598967 +0000 UTC m=+41.268495999" watchObservedRunningTime="2026-11-03 08:39:24.971021763 +0000 UTC m=+41.268918793" Nov 03 08:39:25.002099 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.002065 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-llp7m-mp4tf"] Nov 03 08:39:25.006659 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:25.006626 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0106338_5b18_4bbf_abfe_f2eee3213c8c.slice/crio-5cf9fe8900d2c446a0e01afa796d2f387c7bf1ef01ea9f42c14a1843edf38c8f WatchSource:0}: Error finding container 5cf9fe8900d2c446a0e01afa796d2f387c7bf1ef01ea9f42c14a1843edf38c8f: Status 404 returned error can't find the container with id 5cf9fe8900d2c446a0e01afa796d2f387c7bf1ef01ea9f42c14a1843edf38c8f Nov 03 08:39:25.080821 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.080654 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7"] Nov 03 08:39:25.083050 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:25.083025 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a650ee8_e4e1_4c94_a7da_836ea74de483.slice/crio-f3319e8482b36dc9fc89f37cad48a4addf7aad870d6b24a0e52e3603017b78c5 WatchSource:0}: Error finding container f3319e8482b36dc9fc89f37cad48a4addf7aad870d6b24a0e52e3603017b78c5: Status 404 returned error can't find the container with id f3319e8482b36dc9fc89f37cad48a4addf7aad870d6b24a0e52e3603017b78c5 Nov 03 08:39:25.129903 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.129873 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h"] Nov 03 08:39:25.133392 ip-10-0-131-14 kubenswrapper[2577]: W1103 08:39:25.133362 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcdc2e6_1f23_4da9_a478_0ad2175c5def.slice/crio-bcb10147ce1742dc34bb8a071973c9fadbeafa80ec81da7e11dca8a851403c60 WatchSource:0}: Error finding container bcb10147ce1742dc34bb8a071973c9fadbeafa80ec81da7e11dca8a851403c60: Status 404 returned error can't find the container with id bcb10147ce1742dc34bb8a071973c9fadbeafa80ec81da7e11dca8a851403c60 Nov 03 08:39:25.568194 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.568158 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4cbq5" event={"ID":"f70226b0-5f4c-465c-9ec7-7ceee1bb376f","Type":"ContainerStarted","Data":"fb021124cab16d35cc5a094de1b1ad583728a1086120402ef96be5e9daa474c0"} Nov 03 08:39:25.568838 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.568257 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:25.570366 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.570342 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h" event={"ID":"bfcdc2e6-1f23-4da9-a478-0ad2175c5def","Type":"ContainerStarted","Data":"bcb10147ce1742dc34bb8a071973c9fadbeafa80ec81da7e11dca8a851403c60"} Nov 03 08:39:25.571711 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.571690 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7" event={"ID":"3a650ee8-e4e1-4c94-a7da-836ea74de483","Type":"ContainerStarted","Data":"f3319e8482b36dc9fc89f37cad48a4addf7aad870d6b24a0e52e3603017b78c5"} Nov 03 08:39:25.573365 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.573342 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-llp7m-mp4tf" event={"ID":"e0106338-5b18-4bbf-abfe-f2eee3213c8c","Type":"ContainerStarted","Data":"354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f"} Nov 03 08:39:25.573469 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.573387 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-llp7m-mp4tf" event={"ID":"e0106338-5b18-4bbf-abfe-f2eee3213c8c","Type":"ContainerStarted","Data":"5cf9fe8900d2c446a0e01afa796d2f387c7bf1ef01ea9f42c14a1843edf38c8f"} Nov 03 08:39:25.684067 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.684022 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4cbq5" podStartSLOduration=33.695846014 podStartE2EDuration="40.684008987s" totalImagesPullingTime="6.988162973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:25.683255325 +0000 UTC m=+41.981152354" watchObservedRunningTime="2026-11-03 08:39:25.684008987 +0000 UTC m=+41.981906081" Nov 03 08:39:25.799946 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.799910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:25.800124 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.799959 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:25.800124 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:25.800037 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 08:39:25.800124 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:25.800104 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm podName:19c1fbeb-055b-4c85-aebe-d4856252042a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:27.800085334 +0000 UTC m=+44.097982356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pc6m2" (UID: "19c1fbeb-055b-4c85-aebe-d4856252042a") : configmap "kube-rbac-proxy" not found Nov 03 08:39:25.800296 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:25.800135 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 08:39:25.800296 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:25.800195 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls podName:19c1fbeb-055b-4c85-aebe-d4856252042a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:27.800177245 +0000 UTC m=+44.098074266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pc6m2" (UID: "19c1fbeb-055b-4c85-aebe-d4856252042a") : secret "insights-runtime-extractor-tls" not found Nov 03 08:39:25.800296 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:25.800166 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-llp7m-mp4tf" podStartSLOduration=1.8001481510000001 podStartE2EDuration="1.800148151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:25.7982227 +0000 UTC m=+42.096119731" watchObservedRunningTime="2026-11-03 08:39:25.800148151 +0000 UTC m=+42.098045187" Nov 03 08:39:26.123009 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:26.122974 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:26.123415 ip-10-0-131-14 kubenswrapper[2577]: I1103 08:39:26.123397 2577 scope.go:122] "RemoveContainer" containerID="98480b8ec484ff22c4640ad44514431fb0b225906819ec8a69dfdc4b153fff24" Nov 03 08:39:26.123647 ip-10-0-131-14 kubenswrapper[2577]: E1103 08:39:26.123626 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b2jvc_openshift-console-operator(a21ac76c-2765-417e-832a-32e140173e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podUID="a21ac76c-2765-417e-832a-32e140173e03" Nov 03 08:39:27.011271 ip-10-0-131-14 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 08:39:27.027966 ip-10-0-131-14 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 08:39:27.028136 ip-10-0-131-14 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 08:39:27.028184 ip-10-0-131-14 systemd[1]: kubelet.service: Consumed 3.286s CPU time. Nov 03 08:39:27.042986 ip-10-0-131-14 systemd[1]: Starting Kubernetes Kubelet... Nov 03 08:39:27.105236 ip-10-0-131-14 kubenswrapper[5015]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:39:27.105236 ip-10-0-131-14 kubenswrapper[5015]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 08:39:27.105236 ip-10-0-131-14 kubenswrapper[5015]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:39:27.105236 ip-10-0-131-14 kubenswrapper[5015]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:39:27.107304 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107227 5015 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:27.107304 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107305 5015 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107310 5015 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107313 5015 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107316 5015 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107319 5015 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107321 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107324 5015 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107327 5015 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107329 5015 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107332 5015 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107335 5015 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107338 5015 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107341 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107344 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107346 5015 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107349 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107352 5015 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107354 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107358 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107360 5015 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:27.107378 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107363 5015 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107365 5015 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107368 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107370 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107373 5015 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107375 5015 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107378 5015 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107381 5015 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107384 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107387 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107389 5015 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107392 5015 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107396 5015 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107398 5015 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107401 5015 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107404 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107407 5015 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107409 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107412 5015 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:27.107872 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107414 5015 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107417 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107419 5015 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107421 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107424 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107427 5015 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107429 5015 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107432 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107434 5015 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107437 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107440 5015 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107442 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107445 5015 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107447 5015 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107450 5015 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107453 5015 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107456 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107458 5015 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107461 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107463 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:27.108360 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107466 5015 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107468 5015 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107489 5015 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107493 5015 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107497 5015 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107501 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107504 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107506 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107509 5015 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107512 5015 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107514 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107517 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107519 5015 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107522 5015 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107524 5015 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107527 5015 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107529 5015 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107532 5015 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107534 5015 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:27.108864 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107537 5015 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107540 5015 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107542 5015 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107545 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107548 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107551 5015 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107554 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107556 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107560 5015 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107564 5015 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107567 5015 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107570 5015 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107572 5015 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107575 5015 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107577 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107580 5015 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107582 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107585 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107587 5015 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:27.109317 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107590 5015 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107593 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107596 5015 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107599 5015 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107601 5015 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107604 5015 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107606 5015 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.107609 5015 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108367 5015 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108373 5015 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108377 5015 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108380 5015 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108383 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108386 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108389 5015 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108392 5015 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108394 5015 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108397 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108400 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108402 5015 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:27.109842 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108405 5015 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108408 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108410 5015 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108413 5015 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108415 5015 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108419 5015 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108423 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108426 5015 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108429 5015 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108432 5015 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108435 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108438 5015 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108440 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108443 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108445 5015 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108448 5015 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108450 5015 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108453 5015 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108455 5015 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:27.110325 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108458 5015 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108460 5015 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108462 5015 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108465 5015 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108467 5015 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108470 5015 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108492 5015 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108496 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108502 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108506 5015 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108510 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108513 5015 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108515 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108518 5015 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108521 5015 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108524 5015 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108527 5015 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108529 5015 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108532 5015 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108535 5015 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:27.110796 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108537 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108540 5015 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108543 5015 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108545 5015 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108548 5015 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108550 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108553 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108555 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108558 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108560 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108563 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108565 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108569 5015 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108571 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108574 5015 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108576 5015 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108579 5015 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108581 5015 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108584 5015 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:27.111285 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108588 5015 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108592 5015 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108595 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108599 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108602 5015 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108604 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108607 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108610 5015 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108613 5015 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108616 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108618 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108621 5015 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108623 5015 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108626 5015 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108628 5015 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108631 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108634 5015 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108636 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108640 5015 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108642 5015 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:27.111787 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108645 5015 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108647 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108650 5015 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108653 5015 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108655 5015 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108658 5015 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108660 5015 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108664 5015 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108666 5015 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108669 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108671 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108674 5015 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108676 5015 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108679 5015 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108681 5015 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.108684 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109176 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109182 5015 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109186 5015 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109188 5015 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:27.112278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109191 5015 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109193 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109196 5015 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109202 5015 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109204 5015 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109207 5015 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109209 5015 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109212 5015 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109215 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109217 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109220 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109223 5015 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109225 5015 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109227 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109230 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109232 5015 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109235 5015 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109238 5015 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109240 5015 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:27.112798 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109243 5015 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109245 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109248 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109251 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109253 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109256 5015 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109259 5015 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109261 5015 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109270 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109273 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109276 5015 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109278 5015 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109281 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109283 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109285 5015 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109288 5015 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109292 5015 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109296 5015 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109300 5015 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:27.113300 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109302 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109305 5015 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109308 5015 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109310 5015 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109313 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109316 5015 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109319 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109321 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109324 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109327 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109329 5015 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109332 5015 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109334 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109337 5015 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109340 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109342 5015 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109345 5015 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109348 5015 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109350 5015 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:27.113801 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109353 5015 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109357 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109360 5015 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109365 5015 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109369 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109373 5015 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109376 5015 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109379 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109381 5015 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109384 5015 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109388 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109391 5015 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109393 5015 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109396 5015 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109398 5015 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109401 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109403 5015 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109406 5015 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109409 5015 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109411 5015 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:27.114289 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109414 5015 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109416 5015 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109419 5015 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109422 5015 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109424 5015 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109427 5015 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109430 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109432 5015 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109445 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109448 5015 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109451 5015 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109453 5015 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109456 5015 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109458 5015 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109461 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109463 5015 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109467 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109469 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109488 5015 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109492 5015 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:27.114800 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109497 5015 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109499 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109505 5015 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109509 5015 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.109512 5015 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109616 5015 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109628 5015 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109635 5015 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109639 5015 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109644 5015 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109647 5015 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109651 5015 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109655 5015 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109658 5015 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109662 5015 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109668 5015 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109673 5015 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109678 5015 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109685 5015 flags.go:64] FLAG: --cgroup-root="" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109688 5015 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109691 5015 flags.go:64] FLAG: --client-ca-file="" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109694 5015 flags.go:64] FLAG: --cloud-provider="external" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109697 5015 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 08:39:27.115431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109701 5015 flags.go:64] FLAG: --cluster-domain="" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109704 5015 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109707 5015 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109710 5015 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109713 5015 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109717 5015 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109721 5015 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109724 5015 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109727 5015 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109731 5015 flags.go:64] FLAG: --contention-profiling="false" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109733 5015 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109738 5015 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109741 5015 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109744 5015 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109748 5015 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109751 5015 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109754 5015 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109757 5015 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109760 5015 flags.go:64] FLAG: --enable-server="true" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109763 5015 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109768 5015 flags.go:64] FLAG: --event-burst="100" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109771 5015 flags.go:64] FLAG: --event-qps="50" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109774 5015 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109777 5015 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 08:39:27.116325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109780 5015 flags.go:64] FLAG: --eviction-hard="" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109784 5015 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109787 5015 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109791 5015 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109794 5015 flags.go:64] FLAG: --eviction-soft="" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109797 5015 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109800 5015 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109803 5015 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109806 5015 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109809 5015 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109812 5015 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109815 5015 flags.go:64] FLAG: --feature-gates="" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109819 5015 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109822 5015 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109825 5015 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109831 5015 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109834 5015 flags.go:64] FLAG: --healthz-port="10248" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109837 5015 flags.go:64] FLAG: --help="false" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109840 5015 flags.go:64] FLAG: --hostname-override="ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109845 5015 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109848 5015 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109851 5015 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109855 5015 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 08:39:27.116943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109858 5015 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109861 5015 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109864 5015 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109867 5015 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109869 5015 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109872 5015 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109875 5015 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109878 5015 flags.go:64] FLAG: --kube-reserved="" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109881 5015 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109884 5015 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109887 5015 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109890 5015 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109894 5015 flags.go:64] FLAG: --lock-file="" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109897 5015 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109900 5015 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109903 5015 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109908 5015 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109911 5015 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109914 5015 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109917 5015 flags.go:64] FLAG: --logging-format="text" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109920 5015 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109924 5015 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109926 5015 flags.go:64] FLAG: --manifest-url="" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109929 5015 flags.go:64] FLAG: --manifest-url-header="" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109934 5015 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 08:39:27.117544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109937 5015 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109941 5015 flags.go:64] FLAG: --max-pods="110" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109944 5015 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109948 5015 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109951 5015 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109954 5015 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109957 5015 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109959 5015 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109962 5015 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109969 5015 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109972 5015 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109975 5015 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109978 5015 flags.go:64] FLAG: --pod-cidr="" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109981 5015 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109984 5015 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109987 5015 flags.go:64] FLAG: --pods-per-core="0" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109990 5015 flags.go:64] FLAG: --port="10250" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109993 5015 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.109995 5015 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-009fd1c5f46467c7a" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110000 5015 flags.go:64] FLAG: --qos-reserved="" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110003 5015 flags.go:64] FLAG: --read-only-port="10255" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110006 5015 flags.go:64] FLAG: --register-node="true" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110009 5015 flags.go:64] FLAG: --register-with-taints="" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110013 5015 flags.go:64] FLAG: --registry-burst="10" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110016 5015 flags.go:64] FLAG: --registry-qps="5" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110019 5015 flags.go:64] FLAG: --reserved-cpus="" Nov 03 08:39:27.118134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110022 5015 flags.go:64] FLAG: --reserved-memory="" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110026 5015 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110029 5015 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110032 5015 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110035 5015 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110037 5015 flags.go:64] FLAG: --runonce="false" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110040 5015 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110044 5015 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110047 5015 flags.go:64] FLAG: --seccomp-default="false" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110051 5015 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110054 5015 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110057 5015 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110060 5015 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110063 5015 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110066 5015 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110069 5015 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110072 5015 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110074 5015 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110077 5015 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110080 5015 flags.go:64] FLAG: --system-cgroups="" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110083 5015 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110089 5015 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110092 5015 flags.go:64] FLAG: --tls-cert-file="" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110094 5015 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110098 5015 flags.go:64] FLAG: --tls-min-version="" Nov 03 08:39:27.118872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110102 5015 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110105 5015 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110108 5015 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110111 5015 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110114 5015 flags.go:64] FLAG: --v="2" Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110118 5015 flags.go:64] FLAG: --version="false" Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110122 5015 flags.go:64] FLAG: --vmodule="" Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110126 5015 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110129 5015 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110276 5015 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110281 5015 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110284 5015 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110286 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110289 5015 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110291 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110294 5015 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110301 5015 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110304 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110306 5015 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110309 5015 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110311 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:27.119493 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110314 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110317 5015 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110319 5015 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110322 5015 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110324 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110327 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110329 5015 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110332 5015 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110334 5015 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110339 5015 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110342 5015 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110347 5015 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110350 5015 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110353 5015 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110356 5015 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110358 5015 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110361 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110364 5015 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110366 5015 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:27.120020 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110369 5015 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110372 5015 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110374 5015 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110377 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110379 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110383 5015 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110386 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110389 5015 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110394 5015 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110397 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110399 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110402 5015 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110404 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110407 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110409 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110412 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110414 5015 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110417 5015 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110419 5015 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:27.120517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110422 5015 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110424 5015 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110427 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110429 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110432 5015 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110435 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110452 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110456 5015 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110459 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110463 5015 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110465 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110468 5015 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110488 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110493 5015 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110497 5015 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110500 5015 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110503 5015 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110505 5015 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110508 5015 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110511 5015 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:27.121026 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110513 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110518 5015 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110521 5015 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110524 5015 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110527 5015 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110529 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110532 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110535 5015 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110537 5015 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110540 5015 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110543 5015 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110545 5015 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110547 5015 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110550 5015 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110552 5015 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110555 5015 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110557 5015 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110562 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110565 5015 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110568 5015 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:27.121779 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110570 5015 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110573 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110575 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110578 5015 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110580 5015 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110582 5015 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110585 5015 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110588 5015 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110590 5015 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110592 5015 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110595 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110598 5015 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110601 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110604 5015 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110607 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:27.122290 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110613 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.110640 5015 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110676 5015 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110681 5015 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110684 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110686 5015 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110689 5015 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110691 5015 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110694 5015 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110697 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110700 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110702 5015 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110705 5015 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110708 5015 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110710 5015 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110714 5015 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:27.122683 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110716 5015 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110719 5015 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110721 5015 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110724 5015 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110726 5015 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110729 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110731 5015 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110734 5015 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110737 5015 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110739 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110742 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110744 5015 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110747 5015 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110749 5015 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110753 5015 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110755 5015 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110758 5015 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110766 5015 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110769 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:27.123195 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110772 5015 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110775 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110777 5015 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110780 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110782 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110784 5015 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110787 5015 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110789 5015 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110792 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110794 5015 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110796 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110799 5015 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110803 5015 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110806 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110808 5015 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110811 5015 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110813 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110816 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110818 5015 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110821 5015 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:27.123876 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110824 5015 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110826 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110828 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110831 5015 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110833 5015 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110836 5015 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110840 5015 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110845 5015 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110848 5015 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110851 5015 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110854 5015 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110863 5015 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110866 5015 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110869 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110871 5015 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110874 5015 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110876 5015 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110879 5015 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110881 5015 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:27.124603 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110883 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110886 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110890 5015 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110893 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110896 5015 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110899 5015 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110901 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110904 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110906 5015 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110909 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110911 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110913 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110916 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110918 5015 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110920 5015 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110923 5015 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110925 5015 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110928 5015 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110930 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:27.125076 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110932 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110936 5015 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110939 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110941 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110944 5015 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110946 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110954 5015 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110957 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110959 5015 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110962 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110964 5015 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110967 5015 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110969 5015 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110972 5015 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.110974 5015 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.114861 5015 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.114877 5015 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.114895 5015 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.114899 5015 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115106 5015 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:27.125578 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115118 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115124 5015 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115129 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115133 5015 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115137 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115141 5015 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115146 5015 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115151 5015 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115155 5015 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115160 5015 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115164 5015 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115174 5015 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115178 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115182 5015 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115187 5015 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115191 5015 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115195 5015 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115199 5015 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115204 5015 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:27.126053 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115208 5015 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115212 5015 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115221 5015 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115226 5015 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115230 5015 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115239 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115244 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115249 5015 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115254 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115259 5015 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115263 5015 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115267 5015 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115272 5015 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115277 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115281 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115285 5015 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115289 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115293 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115303 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:27.126517 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115307 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115312 5015 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115316 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115320 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115325 5015 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115329 5015 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115333 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115338 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115343 5015 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115347 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115351 5015 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115361 5015 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115365 5015 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115369 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115373 5015 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115377 5015 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115383 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115390 5015 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115397 5015 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:27.126970 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115403 5015 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115409 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115414 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115419 5015 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115424 5015 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115433 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115439 5015 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115444 5015 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115449 5015 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115453 5015 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115458 5015 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115463 5015 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115468 5015 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115489 5015 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115494 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115499 5015 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115505 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115514 5015 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115519 5015 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115523 5015 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:27.127498 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115527 5015 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115533 5015 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115539 5015 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115544 5015 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115548 5015 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115552 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115556 5015 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115561 5015 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115565 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115574 5015 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115579 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115584 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115589 5015 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115593 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115597 5015 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115601 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115605 5015 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115609 5015 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115613 5015 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115617 5015 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:27.128168 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115621 5015 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115626 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115634 5015 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115638 5015 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115643 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115647 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115651 5015 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.115655 5015 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.115716 5015 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116366 5015 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116378 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116382 5015 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116386 5015 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116389 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116391 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116394 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:27.129081 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116398 5015 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116400 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116403 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116406 5015 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116408 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116411 5015 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116413 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116417 5015 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116419 5015 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116423 5015 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116425 5015 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116428 5015 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116431 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116434 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116439 5015 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116441 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116444 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116447 5015 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116450 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:27.129784 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116452 5015 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116455 5015 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116458 5015 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116461 5015 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116463 5015 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116466 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116469 5015 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116488 5015 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116492 5015 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116496 5015 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116500 5015 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116503 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116506 5015 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116509 5015 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116511 5015 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116514 5015 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116518 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116521 5015 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116523 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116526 5015 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:27.130680 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116528 5015 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116531 5015 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116534 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116536 5015 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116539 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116542 5015 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116545 5015 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116547 5015 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116552 5015 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116556 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116559 5015 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116562 5015 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116564 5015 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116567 5015 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116570 5015 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116572 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116574 5015 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116577 5015 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116580 5015 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116582 5015 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:27.131750 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116585 5015 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116587 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116590 5015 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116592 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116595 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116597 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116600 5015 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116602 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116605 5015 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116608 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116611 5015 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116613 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116617 5015 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116619 5015 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116621 5015 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116624 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116626 5015 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116629 5015 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116631 5015 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:27.132703 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116634 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116636 5015 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116639 5015 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116641 5015 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116644 5015 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116646 5015 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116649 5015 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116652 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116654 5015 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116656 5015 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116659 5015 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116661 5015 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116664 5015 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116666 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116669 5015 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116671 5015 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116673 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116678 5015 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116680 5015 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:27.133661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116683 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116685 5015 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.116717 5015 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116754 5015 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116758 5015 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116761 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116763 5015 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116766 5015 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116768 5015 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116771 5015 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116773 5015 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116777 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116780 5015 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116782 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116785 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116787 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:27.134175 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116789 5015 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116792 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116797 5015 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116800 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116802 5015 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116805 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116808 5015 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116810 5015 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116813 5015 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116815 5015 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116818 5015 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116820 5015 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116823 5015 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116825 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116828 5015 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116830 5015 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116833 5015 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116835 5015 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116838 5015 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:27.134622 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116840 5015 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116842 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116859 5015 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116862 5015 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116865 5015 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116867 5015 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116869 5015 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116872 5015 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116874 5015 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116877 5015 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116879 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116883 5015 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116887 5015 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116890 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116892 5015 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116897 5015 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116900 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116902 5015 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116905 5015 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:27.135073 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116908 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116910 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116913 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116915 5015 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116918 5015 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116920 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116923 5015 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116925 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116928 5015 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116930 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116933 5015 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116935 5015 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116939 5015 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116942 5015 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116945 5015 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116947 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116951 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116953 5015 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116956 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:27.135549 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116959 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116962 5015 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116964 5015 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116967 5015 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116970 5015 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116972 5015 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.116975 5015 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117000 5015 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117004 5015 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117008 5015 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117010 5015 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117013 5015 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117015 5015 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117018 5015 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117021 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117023 5015 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117026 5015 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117028 5015 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117031 5015 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117033 5015 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:27.136038 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117036 5015 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117038 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117041 5015 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117043 5015 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117045 5015 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117048 5015 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117051 5015 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117053 5015 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117056 5015 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117058 5015 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117061 5015 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117066 5015 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117070 5015 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117073 5015 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117075 5015 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:27.117078 5015 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.117135 5015 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.118114 5015 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.118168 5015 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 08:39:27.136613 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.118335 5015 server.go:1014] "Starting client certificate rotation" Nov 03 08:39:27.137148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.118492 5015 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 08:39:27.137148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.118553 5015 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 08:33:45 +0000 UTC" deadline="2028-03-29 01:52:26.686891318 +0000 UTC" Nov 03 08:39:27.137148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.118605 5015 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12281h12m59.568289638s" Nov 03 08:39:27.137148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.118825 5015 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 08:39:27.137148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.119887 5015 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 08:39:27.137148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.121300 5015 log.go:25] "Validated CRI v1 runtime API" Nov 03 08:39:27.137148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.122965 5015 log.go:25] "Validated CRI v1 image API" Nov 03 08:39:27.137148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.123779 5015 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 08:39:27.137148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.125694 5015 fs.go:135] Filesystem UUIDs: map[1defff30-c349-437b-be32-5685024d7217:/dev/nvme0n1p4 42ece542-58a6-4e4d-8086-55b99d712606:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 08:39:27.137411 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.125716 5015 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/016182c16a32739ec7ea9747965e720524005017c05c32a0792ada01d1ad2953/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/016182c16a32739ec7ea9747965e720524005017c05c32a0792ada01d1ad2953/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04a07db94e82003f36e5228103773f0fb3c2aff82c8d90a8da63b3e4fde9910d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04a07db94e82003f36e5228103773f0fb3c2aff82c8d90a8da63b3e4fde9910d/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/087ab6565ab10ccc9d8fd9983e3728e0549df17a15e0e55e01d3201330779bcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/087ab6565ab10ccc9d8fd9983e3728e0549df17a15e0e55e01d3201330779bcd/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0895bf54ec36b69a68bae88e8cdefe6315a189519b23cb005e695488bc9e2991/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0895bf54ec36b69a68bae88e8cdefe6315a189519b23cb005e695488bc9e2991/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/127b3ca309da266ab20064e258f8e4e83965a7722ba6ba0af7430a3178229f7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/127b3ca309da266ab20064e258f8e4e83965a7722ba6ba0af7430a3178229f7b/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13f3d88234344e743fa45f800ae093325a8a8b7c5ada6bb5aa924aa31b9c986f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13f3d88234344e743fa45f800ae093325a8a8b7c5ada6bb5aa924aa31b9c986f/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c55c96a34b1d5c44d14769afe45feb77800e5a27121da34071bb9e1ec64763d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c55c96a34b1d5c44d14769afe45feb77800e5a27121da34071bb9e1ec64763d/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/528a230da6237638550e125b0400d35a8dd30bfb70480c4b4c93a6ab9040d71d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/528a230da6237638550e125b0400d35a8dd30bfb70480c4b4c93a6ab9040d71d/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cf9fe8900d2c446a0e01afa796d2f387c7bf1ef01ea9f42c14a1843edf38c8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cf9fe8900d2c446a0e01afa796d2f387c7bf1ef01ea9f42c14a1843edf38c8f/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5eceb63331ed3385585589b32ec5638ad79823c60bf15484fc86898c82f60468/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5eceb63331ed3385585589b32ec5638ad79823c60bf15484fc86898c82f60468/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/652da50b44684bce010ffe38bdba42f1d68edce3a789c682cfe2608d44ae91eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/652da50b44684bce010ffe38bdba42f1d68edce3a789c682cfe2608d44ae91eb/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96375632bb6d392b66b23f154fdb8ad49256df6c901e8369bc81a63e49fbcbf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96375632bb6d392b66b23f154fdb8ad49256df6c901e8369bc81a63e49fbcbf0/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97393e6231049f704e28e06e0cf4cce23b25376adc82f5f138587bb576d1a834/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97393e6231049f704e28e06e0cf4cce23b25376adc82f5f138587bb576d1a834/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d2b9473c11f04c4cbc7bba86a228f5f5c01ec24ac2bc0159324462bb953b204/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d2b9473c11f04c4cbc7bba86a228f5f5c01ec24ac2bc0159324462bb953b204/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e518abbca50250f10282cc24e25ffbecb12c50ad2979a9a55dc4a797555591e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e518abbca50250f10282cc24e25ffbecb12c50ad2979a9a55dc4a797555591e/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcb10147ce1742dc34bb8a071973c9fadbeafa80ec81da7e11dca8a851403c60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcb10147ce1742dc34bb8a071973c9fadbeafa80ec81da7e11dca8a851403c60/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be3e18905cfc85ed449d206ac748d373ad3e4d5e2e7ad12b44178c4c1657fc2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be3e18905cfc85ed449d206ac748d373ad3e4d5e2e7ad12b44178c4c1657fc2d/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4a924bac8d7adf1065970ff2db52bcea8d3fb3a8206e908d30aa0af07e95b08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4a924bac8d7adf1065970ff2db52bcea8d3fb3a8206e908d30aa0af07e95b08/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2dc18521563827be82c995773f10820ae9dc806a948964a94cf095b6d925772/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2dc18521563827be82c995773f10820ae9dc806a948964a94cf095b6d925772/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db87f4689840a00da5b1ae876f69e4a7be3d81d86d491189367e9bfc7e5d29db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db87f4689840a00da5b1ae876f69e4a7be3d81d86d491189367e9bfc7e5d29db/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc063fcc6f21eebf842e0c3a0c9789bf43cad02feebac8574b72e2cad89eda17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc063fcc6f21eebf842e0c3a0c9789bf43cad02feebac8574b72e2cad89eda17/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e134794b81712988574b0bd790017c2d1f504c5c107d179a506c9c242314eb22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e134794b81712988574b0bd790017c2d1f504c5c107d179a506c9c242314eb22/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3319e8482b36dc9fc89f37cad48a4addf7aad870d6b24a0e52e3603017b78c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3319e8482b36dc9fc89f37cad48a4addf7aad870d6b24a0e52e3603017b78c5/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa9b681cd0ab0633576f0557e9d9631883b42c814c3526bf0b97f88c954c92c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa9b681cd0ab0633576f0557e9d9631883b42c814c3526bf0b97f88c954c92c6/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff30c44d7327fd48a3b59fc2e256027971bcbfca4a60111ac8e45586dee51bf9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff30c44d7327fd48a3b59fc2e256027971bcbfca4a60111ac8e45586dee51bf9/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07f9e882-e987-44b0-a688-45e3c9abec5c/volumes/kubernetes.io~projected/kube-api-access-xccvh:{mountpoint:/var/lib/kubelet/pods/07f9e882-e987-44b0-a688-45e3c9abec5c/volumes/kubernetes.io~projected/kube-api-access-xccvh major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc/volumes/kubernetes.io~projected/kube-api-access-tcv8t:{mountpoint:/var/lib/kubelet/pods/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc/volumes/kubernetes.io~projected/kube-api-access-tcv8t major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1330da31-fcb6-4c7b-9db7-0d4d96262234/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1330da31-fcb6-4c7b-9db7-0d4d96262234/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16cb59f2-9d9f-4904-9fc8-bed9111deecd/volumes/kubernetes.io~projected/kube-api-access-np8js:{mountpoint:/var/lib/kubelet/pods/16cb59f2-9d9f-4904-9fc8-bed9111deecd/volumes/kubernetes.io~projected/kube-api-access-np8js major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16cb59f2-9d9f-4904-9fc8-bed9111deecd/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/16cb59f2-9d9f-4904-9fc8-bed9111deecd/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1959beea-2daf-4e4a-a412-9bc2d35fc277/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1959beea-2daf-4e4a-a412-9bc2d35fc277/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1959beea-2daf-4e4a-a412-9bc2d35fc277/volumes/kubernetes.io~projected/kube-api-access-j55t5:{mountpoint:/var/lib/kubelet/pods/1959beea-2daf-4e4a-a412-9bc2d35fc277/volumes/kubernetes.io~projected/kube-api-access-j55t5 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1959beea-2daf-4e4a-a412-9bc2d35fc277/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1959beea-2daf-4e4a-a412-9bc2d35fc277/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19c1fbeb-055b-4c85-aebe-d4856252042a/volumes/kubernetes.io~projected/kube-api-access-bm8nz:{mountpoint:/var/lib/kubelet/pods/19c1fbeb-055b-4c85-aebe-d4856252042a/volumes/kubernetes.io~projected/kube-api-access-bm8nz major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~projected/kube-api-access-z27j2:{mountpoint:/var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~projected/kube-api-access-z27j2 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bedc7c0-f94a-446e-8029-c3e82609f30f/volumes/kubernetes.io~projected/kube-api-access-6xpgs:{mountpoint:/var/lib/kubelet/pods/1bedc7c0-f94a-446e-8029-c3e82609f30f/volumes/kubernetes.io~projected/kube-api-access-6xpgs major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bedc7c0-f94a-446e-8029-c3e82609f30f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1bedc7c0-f94a-446e-8029-c3e82609f30f/volumes/kubernetes.io~secret/cert major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/204918bd-5b87-4358-b80c-cd6262afb3b0/volumes/kubernetes.io~projected/kube-api-access-tpng7:{mountpoint:/var/lib/kubelet/pods/204918bd-5b87-4358-b80c-cd6262afb3b0/volumes/kubernetes.io~projected/kube-api-access-tpng7 major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22edddd5-3b05-4942-910a-651d537867c8/volumes/kubernetes.io~projected/kube-api-access-4qtzp:{mountpoint:/var/lib/kubelet/pods/22edddd5-3b05-4942-910a-651d537867c8/volumes/kubernetes.io~projected/kube-api-access-4qtzp major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22edddd5-3b05-4942-910a-651d537867c8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/22edddd5-3b05-4942-910a-651d537867c8/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/248b4530-5359-4e26-8309-78d1a7128494/volumes/kubernetes.io~projected/kube-api-access-vtmjl:{mountpoint:/var/lib/kubelet/pods/248b4530-5359-4e26-8309-78d1a7128494/volumes/kubernetes.io~projected/kube-api-access-vtmjl major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25b42294-fcbd-4bc8-8652-9d4d7e762e3d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/25b42294-fcbd-4bc8-8652-9d4d7e762e3d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ed67dd-1988-47af-a5a6-e00ee72236a3/volumes/kubernetes.io~projected/kube-api-access-trbfl:{mountpoint:/var/lib/kubelet/pods/31ed67dd-1988-47af-a5a6-e00ee72236a3/volumes/kubernetes.io~projected/kube-api-access-trbfl major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39df63cd-3b12-4930-8a23-04ed67d961a8/volumes/kubernetes.io~projected/kube-api-access-8v85s:{mountpoint:/var/lib/kubelet/pods/39df63cd-3b12-4930-8a23-04ed67d961a8/volumes/kubernetes.io~projected/kube-api-access-8v85s major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a650ee8-e4e1-4c94-a7da-836ea74de483/volumes/kubernetes.io~projected/kube-api-access-ll7tj:{mountpoint:/var/lib/kubelet/pods/3a650ee8-e4e1-4c94-a7da-836ea74de483/volumes/kubernetes.io~projected/kube-api-access-ll7tj major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57491634-c53c-4571-854c-ece83d3f09a2/volumes/kubernetes.io~projected/kube-api-access-jmrgh:{mountpoint:/var/lib/kubelet/pods/57491634-c53c-4571-854c-ece83d3f09a2/volumes/kubernetes.io~projected/kube-api-access-jmrgh major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587a6bc5-67cb-469b-b4b9-b35da4fb382b/volumes/kubernetes.io~projected/kube-api-access-qlqkf:{mountpoint:/var/lib/kubelet/pods/587a6bc5-67cb-469b-b4b9-b35da4fb382b/volumes/kubernetes.io~projected/kube-api-access-qlqkf major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61655665-1941-4f5f-bc6b-41e51b2b3774/volumes/kubernetes.io~projected/kube-api-access-4mpm6:{mountpoint:/var/lib/kubelet/pods/61655665-1941-4f5f-bc6b-41e51b2b3774/volumes/kubernetes.io~projected/kube-api-access-4mpm6 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63810bfe-e55b-4524-bde4-8f072b4b63a3/volumes/kubernetes.io~projected/kube-api-access-4gnnf:{mountpoint:/var/lib/kubelet/pods/63810bfe-e55b-4524-bde4-8f072b4b63a3/volumes/kubernetes.io~projected/kube-api-access-4gnnf major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70bfc17c-8e18-473a-a8fe-e386112cfaa4/volumes/kubernetes.io~projected/kube-api-access-g5x8t:{mountpoint:/var/lib/kubelet/pods/70bfc17c-8e18-473a-a8fe-e386112cfaa4/volumes/kubernetes.io~projected/kube-api-access-g5x8t major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70bfc17c-8e18-473a-a8fe-e386112cfaa4/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/70bfc17c-8e18-473a-a8fe-e386112cfaa4/volumes/kubernetes.io~secret/default-certificate major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70bfc17c-8e18-473a-a8fe-e386112cfaa4/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/70bfc17c-8e18-473a-a8fe-e386112cfaa4/volumes/kubernetes.io~secret/stats-auth major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c/volumes/kubernetes.io~projected/kube-api-access-th46k:{mountpoint:/var/lib/kubelet/pods/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c/volumes/kubernetes.io~projected/kube-api-access-th46k major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae/volumes/kubernetes.io~projected/kube-api-access-mrtbr:{mountpoint:/var/lib/kubelet/pods/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae/volumes/kubernetes.io~projected/kube-api-access-mrtbr major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83c6ab8c-6c62-4163-a789-1edb75af2ad9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/83c6ab8c-6c62-4163-a789-1edb75af2ad9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83c6ab8c-6c62-4163-a789-1edb75af2ad9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/83c6ab8c-6c62-4163-a789-1edb75af2ad9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83c6ab8c-6c62-4163-a789-1edb75af2ad9/volumes/kubernetes.io~projected/kube-api-access-h2fcj:{mountpoint:/var/lib/kubelet/pods/83c6ab8c-6c62-4163-a789-1edb75af2ad9/volumes/kubernetes.io~projected/kube-api-access-h2fcj major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~projected/kube-api-access-p74hn:{mountpoint:/var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~projected/kube-api-access-p74hn major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a21ac76c-2765-417e-832a-32e140173e03/volumes/kubernetes.io~projected/kube-api-access-x7zqh:{mountpoint:/var/lib/kubelet/pods/a21ac76c-2765-417e-832a-32e140173e03/volumes/kubernetes.io~projected/kube-api-access-x7zqh major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a21ac76c-2765-417e-832a-32e140173e03/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a21ac76c-2765-417e-832a-32e140173e03/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfbee40d-3926-4eae-babe-e598bfae8098/volumes/kubernetes.io~projected/kube-api-access-bhpmt:{mountpoint:/var/lib/kubelet/pods/bfbee40d-3926-4eae-babe-e598bfae8098/volumes/kubernetes.io~projected/kube-api-access-bhpmt major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfcdc2e6-1f23-4da9-a478-0ad2175c5def/volumes/kubernetes.io~projected/kube-api-access-6f27l:{mountpoint:/var/lib/kubelet/pods/bfcdc2e6-1f23-4da9-a478-0ad2175c5def/volumes/kubernetes.io~projected/kube-api-access-6f27l major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a/volumes/kubernetes.io~projected/kube-api-access-m764p:{mountpoint:/var/lib/kubelet/pods/c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a/volumes/kubernetes.io~projected/kube-api-access-m764p major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3a0d3db-7b81-4e10-b240-b665c083bda2/volumes/kubernetes.io~projected/kube-api-access-7rc2j:{mountpoint:/var/lib/kubelet/pods/d3a0d3db-7b81-4e10-b240-b665c083bda2/volumes/kubernetes.io~projected/kube-api-access-7rc2j major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df61dbd1-26b9-4d1b-847c-785687d935ab/volumes/kubernetes.io~projected/kube-api-access-46trz:{mountpoint:/var/lib/kubelet/pods/df61dbd1-26b9-4d1b-847c-785687d935ab/volumes/kubernetes.io~projected/kube-api-access-46trz major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df61dbd1-26b9-4d1b-847c-785687d935ab/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/df61dbd1-26b9-4d1b-847c-785687d935ab/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0106338-5b18-4bbf-abfe-f2eee3213c8c/volumes/kubernetes.io~projected/kube-api-access-wdt5w:{mountpoint:/var/lib/kubelet/pods/e0106338-5b18-4bbf-abfe-f2eee3213c8c/volumes/kubernetes.io~projected/kube-api-access-wdt5w major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f70226b0-5f4c-465c-9ec7-7ceee1bb376f/volumes/kubernetes.io~projected/kube-api-access-4qdq7:{mountpoint:/var/lib/kubelet/pods/f70226b0-5f4c-465c-9ec7-7ceee1bb376f/volumes/kubernetes.io~projected/kube-api-access-4qdq7 major:0 minor:287 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/b79c9341ba9776eced6d7c930ddd023b21bc12e61d9d75e7b13c5a206e637bfa/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e559b6c7f5a3a2cf214a06ac4e916270d36897d69fb8ebb94032f65cc999b32f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/659b3698d2dec2d1e7d14c8ecb83f0a9aae0e0e19bf92617d296c2780c436b32/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/745a5cc490d8eb8eb8a1c09fbd77f0d9900b22075f60e0f1f65bb6efc78c0675/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-115:{mountpoint:/var/lib/containers/storage/overlay/0dee09784790c83e188b9c00f5ebe7ae458b6f90394827d78f38e88b624e31d1/merged major:0 minor:115 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0d3d31f5489dd634f4829f7388a79f6948a0ebfd760491e7c0858e0a75699d9c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/38a3bd441e9cafb61cf96ca04e0cff3b7a3994061975639dfef306dda022d8f1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8a5583323f2db058459443c426d64451971fcd456ff295f3ded929bf78d8ac77/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/81501480a0450a4b4789de230538618007b5730bcded549c0249a9031292b1a5/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/ce2cd5b4bbce087fecf1e5f28bfe1ba729951b3390cc66a64fd061e490095429/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/e954ea398a9a9b9e88c8756c08f30b84d06f99321088593b041ae3d2c3efebe6/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/a8230a474f87a893a47052e8587c5efbc2bb8a15c85c6d2e07cd2612799c5ba4/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6795007c9b83a1aa48eb4dcc81d4b06688df5071a82fc896af61069e33da985a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c3c998153b4a3aecd94f763c2d8443936761225cf8709cfd2c035408425cf1bd/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/757d6a694fa8f248a147f08797370c2958d82a7077cf8fd672a410a81c82da21/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c7f478def9056d9bcd87a27783d36727ba7f7eda9060650f8e62f284ae28ab2f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c545f8e3c6a022a9f77b0325bcd437cd100a9a500d6c5b2fbc962e13d6409d8f/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0747b64fad6613bbfcadea67b024bee65bf6cf962720e4b207231642c8e0db75/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f463e11410c011152bb1c82f5d860d18fac9af869f267c5ff7f4b061de82c2d1/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5fd828d1bf162bc4e9f6cba36ae35085419d328be1a939be46ea04a7a0732cbb/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f6bf6d5d56ba006d733379e28da63b5ca093c7587e2b5f627e5e8684fc4aa7a0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/cfa89ba6a2ba704654c86a944d22f88f871e12d99e6371ad80d7d59b4a2d358c/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/01a3242dcdd7d993ae8ca8f8788a3c6d63c10885907cf0123f0f15b6eb79d380/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/1f0503714b96678fa51df5be413ae716c9ecb24cbefc919e388cf286b48ebeea/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/5d24a961111d8fd59de350d0eed2f7721ed5a29c2e74478109c0b7260eab7340/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f5769172e0b9982c6a987573d1d9704effb9d0209a6e7e53a16de9d25c7a3e47/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/2fd2c82c6d257e628aa1a924cb2d86b78f930bf5816efcb19bbb3fa84ffc340d/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/ba431f09e650620eaf74616d8f4726c761ea590f247d601ab23c52663d1f07b1/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/6addd27b409266e7fdd5563d3e3dac766dcf2659dbff3a5a2dc4821a6a8574b2/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/7726a4337f347fb6ef2ed29a29cdeab16f29088a72ed3764211d7bad021c43b6/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/488e6246c5857ff0426e26d824396ae86272c450c582f5464ed9320a91d96b0e/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/23afccafa290db89a3bf04c57b749a794802b85d0459f42b247a60e5eb77cb59/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/3442917016ddb518e46fe7ef7932f5f90c88fbc64d3a8b438fe1f7b2218692cc/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/2c0aed906c317e453ec267b7debb6523e03f2abf01f9ba0bdd4417bc83130f2b/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/79c38130743f07b3dd2b5b22ac4a6898c38452720f9106230465926421c9b31c/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/1b77e649074c1193d31427c31450ec3308bb623884e1bc8a793954d04afbc404/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/c3961c41aa148513c96a5d1f882ad071f71c9c7fd328b19fde595c5ed876352d/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/3e1749dde93031b7e14825d91b2a86357455b9e0f545e76c8a8751d7ec730445/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/c5a0c2ed6ac8112624f69302cdf94c74d48144c2ab084433d3e59eb83789c289/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/465a1421952a253270fbf880b2a4e08b1b51d2aa24ec0b5ef9614d6485551107/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/e4cf1074a2ecce10d9f381c4fea393ef5e885b46f42943f01a448f5eabc38685/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/b5007a86de77d56fb98a5a828ce10f818ce5ee3440e29f7c2116384393366d15/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/91ea9c127cdd5c93d07b6b0ade475d0470d4db16d2eff21039dc50ef3a413067/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/585e09aab98dd4bf9f5e1c49754daf9bbd968c3e90aba423461921fe786f9139/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/44aa1d499c8a74c5e6856c28ddf93cc8d6c0b5bb07dc9ce2d5002a70652ac629/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/482e96c2e62bb17170415a7844964c6ea8ef20d4db1416e55a7a7acca30e34b9/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/7dda50bb1527a186355337dadd7ad59bb5cabd38c96fcc55b05ca299fcf2e152/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/ba42257873e3e62ccf129dd0af894e921ea5ba0e97c246385022b4858df59a81/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/46e0cfd12c1791b427b5d3773d94aad05170f58316edf56bf95654a827a43607/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a6c0f73146544afe4bbda8df70435bb205df33102858903336dfe0ece4b1fe70/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7fda608e5ecd83357d141db583755a6d12ae03554f6fa744eb4871be277f2b78/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/fb175320b5ea62db740afb82cd096ade9c3e5efbcbed0dbea115420ad339ef04/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/7fda66cae9a992b92fd4eb420ff13930cb28ffc375c4ce43e3f39dea6856f095/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e80cb7c4ab9a786b8b1d6957fb84562f1723fa3f29d6cdba6d82f417e113d253/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/b26cbf7474ee0ff56138bc95fa30bcfc4925921db8e7083a064fbd42fe0cc72f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/cea263da168ff367048d81d665b08b1f5866da39cad7d97cefe56a70b3ab4f12/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a0a7cff529134d9a0865c52ab662bf089563e7eb6cb6aa32056462704ac8852e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ec4e5518f5dc2f0729d596c50f50ba8887a7eb11e8da96805866eefd8a896820/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.135418 5015 manager.go:217] Machine: {Timestamp:2026-11-03 08:39:27.134969077 +0000 UTC m=+0.059911285 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102343 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2baf2f86bb146adae567490cdac778 SystemUUID:ec2baf2f-86bb-146a-dae5-67490cdac778 BootID:4f8d1fc0-9070-4d1e-8efc-a533f29bdc8a Filesystems:[{Device:/run/containers/storage/overlay-containers/016182c16a32739ec7ea9747965e720524005017c05c32a0792ada01d1ad2953/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61655665-1941-4f5f-bc6b-41e51b2b3774/volumes/kubernetes.io~projected/kube-api-access-4mpm6 DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a21ac76c-2765-417e-832a-32e140173e03/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22edddd5-3b05-4942-910a-651d537867c8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a21ac76c-2765-417e-832a-32e140173e03/volumes/kubernetes.io~projected/kube-api-access-x7zqh DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70bfc17c-8e18-473a-a8fe-e386112cfaa4/volumes/kubernetes.io~projected/kube-api-access-g5x8t DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a650ee8-e4e1-4c94-a7da-836ea74de483/volumes/kubernetes.io~projected/kube-api-access-ll7tj DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c55c96a34b1d5c44d14769afe45feb77800e5a27121da34071bb9e1ec64763d/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16cb59f2-9d9f-4904-9fc8-bed9111deecd/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/652da50b44684bce010ffe38bdba42f1d68edce3a789c682cfe2608d44ae91eb/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3319e8482b36dc9fc89f37cad48a4addf7aad870d6b24a0e52e3603017b78c5/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07f9e882-e987-44b0-a688-45e3c9abec5c/volumes/kubernetes.io~projected/kube-api-access-xccvh DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d3a0d3db-7b81-4e10-b240-b665c083bda2/volumes/kubernetes.io~projected/kube-api-access-7rc2j DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25b42294-fcbd-4bc8-8652-9d4d7e762e3d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19c1fbeb-055b-4c85-aebe-d4856252042a/volumes/kubernetes.io~projected/kube-api-access-bm8nz DeviceMajor:0 DeviceMinor:333 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db87f4689840a00da5b1ae876f69e4a7be3d81d86d491189367e9bfc7e5d29db/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e134794b81712988574b0bd790017c2d1f504c5c107d179a506c9c242314eb22/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2dc18521563827be82c995773f10820ae9dc806a948964a94cf095b6d925772/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1959beea-2daf-4e4a-a412-9bc2d35fc277/volumes/kubernetes.io~projected/kube-api-access-j55t5 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d2b9473c11f04c4cbc7bba86a228f5f5c01ec24ac2bc0159324462bb953b204/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63810bfe-e55b-4524-bde4-8f072b4b63a3/volumes/kubernetes.io~projected/kube-api-access-4gnnf DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e518abbca50250f10282cc24e25ffbecb12c50ad2979a9a55dc4a797555591e/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be3e18905cfc85ed449d206ac748d373ad3e4d5e2e7ad12b44178c4c1657fc2d/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff30c44d7327fd48a3b59fc2e256027971bcbfca4a60111ac8e45586dee51bf9/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1330da31-fcb6-4c7b-9db7-0d4d96262234/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/83c6ab8c-6c62-4163-a789-1edb75af2ad9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c/volumes/kubernetes.io~projected/kube-api-access-th46k DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f70226b0-5f4c-465c-9ec7-7ceee1bb376f/volumes/kubernetes.io~projected/kube-api-access-4qdq7 DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/83c6ab8c-6c62-4163-a789-1edb75af2ad9/volumes/kubernetes.io~projected/kube-api-access-h2fcj DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04a07db94e82003f36e5228103773f0fb3c2aff82c8d90a8da63b3e4fde9910d/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83c6ab8c-6c62-4163-a789-1edb75af2ad9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1bedc7c0-f94a-446e-8029-c3e82609f30f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfcdc2e6-1f23-4da9-a478-0ad2175c5def/volumes/kubernetes.io~projected/kube-api-access-6f27l DeviceMajor:0 DeviceMinor:337 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/528a230da6237638550e125b0400d35a8dd30bfb70480c4b4c93a6ab9040d71d/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a/volumes/kubernetes.io~projected/kube-api-access-m764p DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5eceb63331ed3385585589b32ec5638ad79823c60bf15484fc86898c82f60468/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97393e6231049f704e28e06e0cf4cce23b25376adc82f5f138587bb576d1a834/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc/volumes/kubernetes.io~projected/kube-api-access-tcv8t DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-115 DeviceMajor:0 DeviceMinor:115 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df61dbd1-26b9-4d1b-847c-785687d935ab/volumes/kubernetes.io~projected/kube-api-access-46trz DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~projected/kube-api-access-z27j2 DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0106338-5b18-4bbf-abfe-f2eee3213c8c/volumes/kubernetes.io~projected/kube-api-access-wdt5w DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc063fcc6f21eebf842e0c3a0c9789bf43cad02feebac8574b72e2cad89eda17/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/587a6bc5-67cb-469b-b4b9-b35da4fb382b/volumes/kubernetes.io~projected/kube-api-access-qlqkf DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70bfc17c-8e18-473a-a8fe-e386112cfaa4/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/204918bd-5b87-4358-b80c-cd6262afb3b0/volumes/kubernetes.io~projected/kube-api-access-tpng7 DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~projected/kube-api-access-p74hn DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cf9fe8900d2c446a0e01afa796d2f387c7bf1ef01ea9f42c14a1843edf38c8f/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/248b4530-5359-4e26-8309-78d1a7128494/volumes/kubernetes.io~projected/kube-api-access-vtmjl DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/087ab6565ab10ccc9d8fd9983e3728e0549df17a15e0e55e01d3201330779bcd/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16cb59f2-9d9f-4904-9fc8-bed9111deecd/volumes/kubernetes.io~projected/kube-api-access-np8js DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13f3d88234344e743fa45f800ae093325a8a8b7c5ada6bb5aa924aa31b9c986f/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1959beea-2daf-4e4a-a412-9bc2d35fc277/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/39df63cd-3b12-4930-8a23-04ed67d961a8/volumes/kubernetes.io~projected/kube-api-access-8v85s DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcb10147ce1742dc34bb8a071973c9fadbeafa80ec81da7e11dca8a851403c60/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae/volumes/kubernetes.io~projected/kube-api-access-mrtbr DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70bfc17c-8e18-473a-a8fe-e386112cfaa4/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/127b3ca309da266ab20064e258f8e4e83965a7722ba6ba0af7430a3178229f7b/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfbee40d-3926-4eae-babe-e598bfae8098/volumes/kubernetes.io~projected/kube-api-access-bhpmt DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57491634-c53c-4571-854c-ece83d3f09a2/volumes/kubernetes.io~projected/kube-api-access-jmrgh DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22edddd5-3b05-4942-910a-651d537867c8/volumes/kubernetes.io~projected/kube-api-access-4qtzp DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1959beea-2daf-4e4a-a412-9bc2d35fc277/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df61dbd1-26b9-4d1b-847c-785687d935ab/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1bedc7c0-f94a-446e-8029-c3e82609f30f/volumes/kubernetes.io~projected/kube-api-access-6xpgs DeviceMajor:0 DeviceMinor:239 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa9b681cd0ab0633576f0557e9d9631883b42c814c3526bf0b97f88c954c92c6/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31ed67dd-1988-47af-a5a6-e00ee72236a3/volumes/kubernetes.io~projected/kube-api-access-trbfl DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4a924bac8d7adf1065970ff2db52bcea8d3fb3a8206e908d30aa0af07e95b08/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96375632bb6d392b66b23f154fdb8ad49256df6c901e8369bc81a63e49fbcbf0/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0895bf54ec36b69a68bae88e8cdefe6315a189519b23cb005e695488bc9e2991/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:016182c16a32739 MacAddress:be:5d:1b:f7:8f:3c Speed:10000 Mtu:8901} {Name:087ab6565ab10cc MacAddress:aa:fa:1a:5c:ad:bb Speed:10000 Mtu:8901} {Name:0895bf54ec36b69 MacAddress:26:02:36:0e:f2:ed Speed:10000 Mtu:8901} {Name:127b3ca309da266 MacAddress:1a:aa:54:36:48:47 Speed:10000 Mtu:8901} {Name:13f3d88234344e7 MacAddress:d6:ce:e3:2e:3d:08 Speed:10000 Mtu:8901} {Name:5cf9fe8900d2c44 MacAddress:66:48:a6:39:9c:c9 Speed:10000 Mtu:8901} {Name:5eceb63331ed338 MacAddress:46:a5:d8:a4:bf:ab Speed:10000 Mtu:8901} {Name:652da50b44684bc MacAddress:f6:a5:cf:c5:84:8e Speed:10000 Mtu:8901} {Name:97393e6231049f7 MacAddress:76:b2:f9:0a:d4:04 Speed:10000 Mtu:8901} {Name:9e518abbca50250 MacAddress:5a:36:82:ef:ad:96 Speed:10000 Mtu:8901} {Name:bcb10147ce1742d MacAddress:8e:f2:4e:a1:e0:c0 Speed:10000 Mtu:8901} {Name:be3e18905cfc85e MacAddress:c2:cd:28:90:a7:1a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:12:e0:7e:3b:19 Speed:0 Mtu:9001} {Name:br-int MacAddress:36:f8:f6:e4:0e:9c Speed:0 Mtu:8901} {Name:d2dc18521563827 MacAddress:ea:72:14:61:47:a0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:12:e0:7e:3b:19 Speed:0 Mtu:9001} {Name:f3319e8482b36dc MacAddress:ae:a4:6d:fa:96:bb Speed:10000 Mtu:8901} {Name:ff30c44d7327fd4 MacAddress:52:0d:1e:af:34:d1 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2a:1c:2b:ce:ca:63 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fa:fa:27:b0:c1:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.135810 5015 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.135855 5015 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136120 5015 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136137 5015 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-14.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136258 5015 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136267 5015 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136276 5015 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136295 5015 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136313 5015 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136453 5015 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136593 5015 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136626 5015 kubelet.go:493] "Attempting to sync node with API server" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136643 5015 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136663 5015 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136676 5015 kubelet.go:405] "Adding apiserver pod source" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.136689 5015 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.137650 5015 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 08:39:27.137674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.137664 5015 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 08:39:27.138341 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.138149 5015 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 08:39:27.150205 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150189 5015 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 08:39:27.150205 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150204 5015 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 08:39:27.150318 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150302 5015 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 08:39:27.150516 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150505 5015 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 08:39:27.150547 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150535 5015 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 08:39:27.150633 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150625 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 08:39:27.150662 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150643 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 08:39:27.150662 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150652 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 08:39:27.150662 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150660 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 08:39:27.150740 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150665 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 08:39:27.150740 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150672 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 08:39:27.150740 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150678 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 08:39:27.150740 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150683 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 08:39:27.150740 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150689 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 08:39:27.150740 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150695 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 08:39:27.150740 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150708 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 08:39:27.150740 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150720 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 08:39:27.150942 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150744 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 08:39:27.150942 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.150753 5015 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 08:39:27.154293 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.154280 5015 server.go:1263] "Started kubelet" Nov 03 08:39:27.154387 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.154344 5015 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 08:39:27.154442 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.154365 5015 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 08:39:27.154933 ip-10-0-131-14 systemd[1]: Started Kubernetes Kubelet. Nov 03 08:39:27.155068 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.154970 5015 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 08:39:27.155625 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.155445 5015 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 08:39:27.156850 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.156835 5015 server.go:317] "Adding debug handlers to kubelet server" Nov 03 08:39:27.159784 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.159763 5015 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 08:39:27.159865 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.159817 5015 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 08:39:27.159932 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.159868 5015 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 08:39:27.159932 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.159881 5015 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 08:39:27.159932 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.159883 5015 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 08:33:46 +0000 UTC" deadline="2028-03-30 08:16:38.227526287 +0000 UTC" Nov 03 08:39:27.159932 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.159922 5015 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12311h37m11.06760838s" Nov 03 08:39:27.160126 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:27.159955 5015 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-14.us-west-1.compute.internal\" not found" Nov 03 08:39:27.161253 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.160960 5015 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 08:39:27.161253 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.161179 5015 factory.go:55] Registering systemd factory Nov 03 08:39:27.161253 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.161200 5015 factory.go:223] Registration of the systemd container factory successfully Nov 03 08:39:27.164194 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164176 5015 factory.go:153] Registering CRI-O factory Nov 03 08:39:27.164194 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164196 5015 factory.go:223] Registration of the crio container factory successfully Nov 03 08:39:27.164334 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164248 5015 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 08:39:27.164334 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164267 5015 factory.go:103] Registering Raw factory Nov 03 08:39:27.164334 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164281 5015 manager.go:1199] Started watching for new ooms in manager Nov 03 08:39:27.164628 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164617 5015 manager.go:319] Starting recovery of all containers Nov 03 08:39:27.164764 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164720 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" volumeName="kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-certificates" seLinuxMountContext="" Nov 03 08:39:27.164801 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164770 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ed67dd-1988-47af-a5a6-e00ee72236a3" volumeName="kubernetes.io/projected/31ed67dd-1988-47af-a5a6-e00ee72236a3-kube-api-access-trbfl" seLinuxMountContext="" Nov 03 08:39:27.164801 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164783 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587a6bc5-67cb-469b-b4b9-b35da4fb382b" volumeName="kubernetes.io/projected/587a6bc5-67cb-469b-b4b9-b35da4fb382b-kube-api-access-qlqkf" seLinuxMountContext="" Nov 03 08:39:27.164801 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164797 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df61dbd1-26b9-4d1b-847c-785687d935ab" volumeName="kubernetes.io/projected/df61dbd1-26b9-4d1b-847c-785687d935ab-kube-api-access-46trz" seLinuxMountContext="" Nov 03 08:39:27.164899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164811 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1959beea-2daf-4e4a-a412-9bc2d35fc277" volumeName="kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-script-lib" seLinuxMountContext="" Nov 03 08:39:27.164899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164824 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70bfc17c-8e18-473a-a8fe-e386112cfaa4" volumeName="kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-stats-auth" seLinuxMountContext="" Nov 03 08:39:27.164899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164836 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bebbc2e-c649-4f7f-bef3-0a8f158d6cae" volumeName="kubernetes.io/projected/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-kube-api-access-mrtbr" seLinuxMountContext="" Nov 03 08:39:27.164899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164848 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83c6ab8c-6c62-4163-a789-1edb75af2ad9" volumeName="kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-tmp" seLinuxMountContext="" Nov 03 08:39:27.164899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164863 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0e56142-8762-447d-af25-d0ba1b09b530" volumeName="kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-image-registry-private-configuration" seLinuxMountContext="" Nov 03 08:39:27.164899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164874 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df61dbd1-26b9-4d1b-847c-785687d935ab" volumeName="kubernetes.io/configmap/df61dbd1-26b9-4d1b-847c-785687d935ab-config" seLinuxMountContext="" Nov 03 08:39:27.164899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164886 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07f9e882-e987-44b0-a688-45e3c9abec5c" volumeName="kubernetes.io/configmap/07f9e882-e987-44b0-a688-45e3c9abec5c-iptables-alerter-script" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164903 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" volumeName="kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-installation-pull-secrets" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164919 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="248b4530-5359-4e26-8309-78d1a7128494" volumeName="kubernetes.io/projected/248b4530-5359-4e26-8309-78d1a7128494-kube-api-access-vtmjl" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164930 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83c6ab8c-6c62-4163-a789-1edb75af2ad9" volumeName="kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-tuned" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164955 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="204918bd-5b87-4358-b80c-cd6262afb3b0" volumeName="kubernetes.io/projected/204918bd-5b87-4358-b80c-cd6262afb3b0-kube-api-access-tpng7" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164967 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" volumeName="kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164978 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" volumeName="kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-binary-copy" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.164990 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" volumeName="kubernetes.io/projected/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-kube-api-access-tcv8t" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165014 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" volumeName="kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-trusted-ca" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165025 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" volumeName="kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-image-registry-private-configuration" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165037 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bedc7c0-f94a-446e-8029-c3e82609f30f" volumeName="kubernetes.io/projected/1bedc7c0-f94a-446e-8029-c3e82609f30f-kube-api-access-6xpgs" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165049 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfbee40d-3926-4eae-babe-e598bfae8098" volumeName="kubernetes.io/projected/bfbee40d-3926-4eae-babe-e598bfae8098-kube-api-access-bhpmt" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165061 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70bfc17c-8e18-473a-a8fe-e386112cfaa4" volumeName="kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-default-certificate" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165072 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bebbc2e-c649-4f7f-bef3-0a8f158d6cae" volumeName="kubernetes.io/configmap/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-config" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165084 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1330da31-fcb6-4c7b-9db7-0d4d96262234" volumeName="kubernetes.io/configmap/1330da31-fcb6-4c7b-9db7-0d4d96262234-konnectivity-ca" seLinuxMountContext="" Nov 03 08:39:27.165094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165096 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1330da31-fcb6-4c7b-9db7-0d4d96262234" volumeName="kubernetes.io/secret/1330da31-fcb6-4c7b-9db7-0d4d96262234-agent-certs" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165108 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0e56142-8762-447d-af25-d0ba1b09b530" volumeName="kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-installation-pull-secrets" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165120 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0106338-5b18-4bbf-abfe-f2eee3213c8c" volumeName="kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-trusted-ca-bundle" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165139 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07f9e882-e987-44b0-a688-45e3c9abec5c" volumeName="kubernetes.io/projected/07f9e882-e987-44b0-a688-45e3c9abec5c-kube-api-access-xccvh" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165151 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16cb59f2-9d9f-4904-9fc8-bed9111deecd" volumeName="kubernetes.io/projected/16cb59f2-9d9f-4904-9fc8-bed9111deecd-kube-api-access-np8js" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165168 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" volumeName="kubernetes.io/empty-dir/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-ca-trust-extracted" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165180 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22edddd5-3b05-4942-910a-651d537867c8" volumeName="kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-tmp" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165192 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="283e4bdb-5662-4b18-8775-7c86dd4323f3" volumeName="kubernetes.io/configmap/283e4bdb-5662-4b18-8775-7c86dd4323f3-nginx-conf" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165205 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0e56142-8762-447d-af25-d0ba1b09b530" volumeName="kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-registry-certificates" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165215 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21ac76c-2765-417e-832a-32e140173e03" volumeName="kubernetes.io/secret/a21ac76c-2765-417e-832a-32e140173e03-serving-cert" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165227 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df61dbd1-26b9-4d1b-847c-785687d935ab" volumeName="kubernetes.io/secret/df61dbd1-26b9-4d1b-847c-785687d935ab-serving-cert" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165242 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1959beea-2daf-4e4a-a412-9bc2d35fc277" volumeName="kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-env-overrides" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165253 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61655665-1941-4f5f-bc6b-41e51b2b3774" volumeName="kubernetes.io/projected/61655665-1941-4f5f-bc6b-41e51b2b3774-kube-api-access-4mpm6" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165266 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfcdc2e6-1f23-4da9-a478-0ad2175c5def" volumeName="kubernetes.io/projected/bfcdc2e6-1f23-4da9-a478-0ad2175c5def-kube-api-access-6f27l" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165279 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a0d3db-7b81-4e10-b240-b665c083bda2" volumeName="kubernetes.io/projected/d3a0d3db-7b81-4e10-b240-b665c083bda2-kube-api-access-7rc2j" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165291 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0106338-5b18-4bbf-abfe-f2eee3213c8c" volumeName="kubernetes.io/projected/e0106338-5b18-4bbf-abfe-f2eee3213c8c-kube-api-access-wdt5w" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165339 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f70226b0-5f4c-465c-9ec7-7ceee1bb376f" volumeName="kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165355 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bedc7c0-f94a-446e-8029-c3e82609f30f" volumeName="kubernetes.io/secret/1bedc7c0-f94a-446e-8029-c3e82609f30f-cert" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165369 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83c6ab8c-6c62-4163-a789-1edb75af2ad9" volumeName="kubernetes.io/projected/83c6ab8c-6c62-4163-a789-1edb75af2ad9-kube-api-access-h2fcj" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165381 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0e56142-8762-447d-af25-d0ba1b09b530" volumeName="kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-kube-api-access-p74hn" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165392 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21ac76c-2765-417e-832a-32e140173e03" volumeName="kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-config" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165406 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" volumeName="kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-bound-sa-token" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165417 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" volumeName="kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165432 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1959beea-2daf-4e4a-a412-9bc2d35fc277" volumeName="kubernetes.io/secret/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165454 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63810bfe-e55b-4524-bde4-8f072b4b63a3" volumeName="kubernetes.io/projected/63810bfe-e55b-4524-bde4-8f072b4b63a3-kube-api-access-4gnnf" seLinuxMountContext="" Nov 03 08:39:27.165549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165465 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1959beea-2daf-4e4a-a412-9bc2d35fc277" volumeName="kubernetes.io/projected/1959beea-2daf-4e4a-a412-9bc2d35fc277-kube-api-access-j55t5" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165564 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22edddd5-3b05-4942-910a-651d537867c8" volumeName="kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-snapshots" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165576 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587a6bc5-67cb-469b-b4b9-b35da4fb382b" volumeName="kubernetes.io/empty-dir/587a6bc5-67cb-469b-b4b9-b35da4fb382b-tmp-dir" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165587 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70bfc17c-8e18-473a-a8fe-e386112cfaa4" volumeName="kubernetes.io/projected/70bfc17c-8e18-473a-a8fe-e386112cfaa4-kube-api-access-g5x8t" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165602 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0e56142-8762-447d-af25-d0ba1b09b530" volumeName="kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-trusted-ca" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165613 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57491634-c53c-4571-854c-ece83d3f09a2" volumeName="kubernetes.io/projected/57491634-c53c-4571-854c-ece83d3f09a2-kube-api-access-jmrgh" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165636 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0e56142-8762-447d-af25-d0ba1b09b530" volumeName="kubernetes.io/empty-dir/a0e56142-8762-447d-af25-d0ba1b09b530-ca-trust-extracted" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165649 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21ac76c-2765-417e-832a-32e140173e03" volumeName="kubernetes.io/projected/a21ac76c-2765-417e-832a-32e140173e03-kube-api-access-x7zqh" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165661 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39df63cd-3b12-4930-8a23-04ed67d961a8" volumeName="kubernetes.io/projected/39df63cd-3b12-4930-8a23-04ed67d961a8-kube-api-access-8v85s" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165674 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d5c7ae-f8aa-4300-b5a6-c76273b66b3c" volumeName="kubernetes.io/projected/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-kube-api-access-th46k" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165689 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16cb59f2-9d9f-4904-9fc8-bed9111deecd" volumeName="kubernetes.io/secret/16cb59f2-9d9f-4904-9fc8-bed9111deecd-serving-cert" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165700 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0e56142-8762-447d-af25-d0ba1b09b530" volumeName="kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-bound-sa-token" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165712 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21ac76c-2765-417e-832a-32e140173e03" volumeName="kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-trusted-ca" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165724 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a" volumeName="kubernetes.io/projected/c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a-kube-api-access-m764p" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165735 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0106338-5b18-4bbf-abfe-f2eee3213c8c" volumeName="kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-service-ca-bundle" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165747 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0106338-5b18-4bbf-abfe-f2eee3213c8c" volumeName="kubernetes.io/empty-dir/e0106338-5b18-4bbf-abfe-f2eee3213c8c-archives-path" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165760 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57491634-c53c-4571-854c-ece83d3f09a2" volumeName="kubernetes.io/configmap/57491634-c53c-4571-854c-ece83d3f09a2-telemetry-config" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165771 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16cb59f2-9d9f-4904-9fc8-bed9111deecd" volumeName="kubernetes.io/configmap/16cb59f2-9d9f-4904-9fc8-bed9111deecd-config" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165785 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1959beea-2daf-4e4a-a412-9bc2d35fc277" volumeName="kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-config" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165797 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c1fbeb-055b-4c85-aebe-d4856252042a" volumeName="kubernetes.io/empty-dir/19c1fbeb-055b-4c85-aebe-d4856252042a-data-volume" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165807 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22edddd5-3b05-4942-910a-651d537867c8" volumeName="kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-trusted-ca-bundle" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165819 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22edddd5-3b05-4942-910a-651d537867c8" volumeName="kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-service-ca-bundle" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165832 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39df63cd-3b12-4930-8a23-04ed67d961a8" volumeName="kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-cni-binary-copy" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165844 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c1fbeb-055b-4c85-aebe-d4856252042a" volumeName="kubernetes.io/projected/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-api-access-bm8nz" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165858 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22edddd5-3b05-4942-910a-651d537867c8" volumeName="kubernetes.io/projected/22edddd5-3b05-4942-910a-651d537867c8-kube-api-access-4qtzp" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165869 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25b42294-fcbd-4bc8-8652-9d4d7e762e3d" volumeName="kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165881 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d5c7ae-f8aa-4300-b5a6-c76273b66b3c" volumeName="kubernetes.io/empty-dir/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-tmp-dir" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165904 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a0d3db-7b81-4e10-b240-b665c083bda2" volumeName="kubernetes.io/configmap/d3a0d3db-7b81-4e10-b240-b665c083bda2-serviceca" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165917 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" volumeName="kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-kube-api-access-z27j2" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165929 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22edddd5-3b05-4942-910a-651d537867c8" volumeName="kubernetes.io/secret/22edddd5-3b05-4942-910a-651d537867c8-serving-cert" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165941 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39df63cd-3b12-4930-8a23-04ed67d961a8" volumeName="kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-daemon-config" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165951 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a650ee8-e4e1-4c94-a7da-836ea74de483" volumeName="kubernetes.io/projected/3a650ee8-e4e1-4c94-a7da-836ea74de483-kube-api-access-ll7tj" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165963 5015 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587a6bc5-67cb-469b-b4b9-b35da4fb382b" volumeName="kubernetes.io/configmap/587a6bc5-67cb-469b-b4b9-b35da4fb382b-config-volume" seLinuxMountContext="" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165976 5015 reconstruct.go:97] "Volume reconstruction finished" Nov 03 08:39:27.166158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.165984 5015 reconciler.go:29] "Reconciler: start to sync state" Nov 03 08:39:27.174103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.174050 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/1.log" Nov 03 08:39:27.176838 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.176817 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/1.log" Nov 03 08:39:27.194636 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.194609 5015 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 08:39:27.196035 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.196011 5015 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 08:39:27.196035 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.196036 5015 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 08:39:27.196168 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.196058 5015 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 08:39:27.196168 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:27.196090 5015 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 08:39:27.204224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.204203 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/1.log" Nov 03 08:39:27.204329 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.204232 5015 generic.go:356] "Generic (PLEG): container finished" podID="a21ac76c-2765-417e-832a-32e140173e03" containerID="98480b8ec484ff22c4640ad44514431fb0b225906819ec8a69dfdc4b153fff24" exitCode=255 Nov 03 08:39:27.233234 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.233193 5015 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="30e2e6d64b3181c88fe61da946b3a38e5aa1bcf92c745e94e0b51b8bdb7b7422" exitCode=0 Nov 03 08:39:27.233234 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.233224 5015 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="cffe0ae561eab3368afbc538178e223e7a5b4aeda1d3a57fd240698031d8f54f" exitCode=0 Nov 03 08:39:27.233234 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.233233 5015 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="4f266ce650eacbe66d84c6ef92e19b97d43a88dffb646d70468b827bdd3f5aef" exitCode=0 Nov 03 08:39:27.233502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.233246 5015 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="f88ad47e1fc20a89b9a4693843555a1ea08513097824b3ddb15079a2e56cc7ec" exitCode=0 Nov 03 08:39:27.233502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.233257 5015 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="dcb483f3e40d8ec49fbc68a28a76f7f53601363125b0527b64ef78c43c07680e" exitCode=0 Nov 03 08:39:27.233502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.233268 5015 generic.go:356] "Generic (PLEG): container finished" podID="0893a4fe-c6b7-42e5-8e5c-8d179dfefedc" containerID="31b9363fc26d3d999dbe30e48dcd58c9822c7dfa65a24b364b164cd6beff5fa3" exitCode=0 Nov 03 08:39:27.241002 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.240973 5015 generic.go:356] "Generic (PLEG): container finished" podID="0b188283236722665eb092553b8ae03f" containerID="6ad903de0367c48d3dbc3e66dc042aaed6bc3ff78907c0020bafa27cab41410f" exitCode=0 Nov 03 08:39:27.260044 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:27.260018 5015 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-14.us-west-1.compute.internal\" not found" Nov 03 08:39:27.260607 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.260590 5015 manager.go:324] Recovery completed Nov 03 08:39:27.278155 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.278109 5015 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:27.279282 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.279267 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:27.279353 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.279295 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:27.279353 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.279306 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:27.283087 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283061 5015 cpu_manager.go:226] "Starting" policy="none" Nov 03 08:39:27.283087 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283084 5015 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 08:39:27.283239 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283105 5015 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 08:39:27.283286 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283275 5015 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 08:39:27.283329 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283290 5015 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 08:39:27.283405 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283390 5015 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 08:39:27.283449 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283407 5015 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 08:39:27.283449 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283415 5015 policy_none.go:50] "Start" Nov 03 08:39:27.283449 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283434 5015 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 08:39:27.283449 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283447 5015 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 08:39:27.283644 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283466 5015 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:27.283715 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283668 5015 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 08:39:27.283715 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283676 5015 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 08:39:27.283715 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.283698 5015 policy_none.go:44] "Start" Nov 03 08:39:27.284775 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.284757 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:27.284867 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.284793 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:27.284867 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.284808 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:27.288580 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.288549 5015 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 08:39:27.291182 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.291167 5015 manager.go:341] "Starting Device Plugin manager" Nov 03 08:39:27.291232 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:27.291199 5015 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 08:39:27.291232 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.291213 5015 server.go:87] "Starting device plugin registration server" Nov 03 08:39:27.291562 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.291549 5015 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 08:39:27.291603 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.291565 5015 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 08:39:27.291673 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.291652 5015 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 08:39:27.291808 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.291760 5015 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 08:39:27.291877 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.291809 5015 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 08:39:27.291877 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.291816 5015 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 08:39:27.291961 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.291868 5015 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T08:39:27.291796668Z","UUID":"dfd79963-d8e6-4a31-bd90-70096d8b6743","Handler":null,"Name":"","Endpoint":""} Nov 03 08:39:27.293142 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.293125 5015 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 08:39:27.293239 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.293162 5015 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 08:39:27.293386 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:27.293363 5015 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 08:39:27.293539 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:27.293405 5015 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-14.us-west-1.compute.internal\" not found" Nov 03 08:39:27.296462 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.296209 5015 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal"] Nov 03 08:39:27.296602 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.296586 5015 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:27.297402 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.297388 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:27.297524 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.297410 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:27.297524 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.297423 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:27.297790 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.297776 5015 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:27.297928 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.297902 5015 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:27.298682 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.298663 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:27.298781 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.298693 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:27.298781 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.298703 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:27.298781 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.298730 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:27.298781 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.298746 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:27.298781 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.298708 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:27.298994 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:27.298980 5015 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-14.us-west-1.compute.internal\" not found" node="ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.299278 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.299131 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" event={"ID":"0b188283236722665eb092553b8ae03f","Type":"ContainerStarted","Data":"770099cca10f3c04381449ea6b93fce72606da522fad12fe5ed1ae1316ddc62e"} Nov 03 08:39:27.299278 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.299178 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" event={"ID":"0b188283236722665eb092553b8ae03f","Type":"ContainerDied","Data":"6ad903de0367c48d3dbc3e66dc042aaed6bc3ff78907c0020bafa27cab41410f"} Nov 03 08:39:27.299278 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.299201 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" event={"ID":"0b188283236722665eb092553b8ae03f","Type":"ContainerStarted","Data":"04a07db94e82003f36e5228103773f0fb3c2aff82c8d90a8da63b3e4fde9910d"} Nov 03 08:39:27.299278 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.299232 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" event={"ID":"6652f123a319d675474cd7afd732c542","Type":"ContainerStarted","Data":"d809d0321bdc230cb3eae37baceb04ebb96210124e7663c884c5e6e04954c023"} Nov 03 08:39:27.299278 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.299248 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" event={"ID":"6652f123a319d675474cd7afd732c542","Type":"ContainerStarted","Data":"3c55c96a34b1d5c44d14769afe45feb77800e5a27121da34071bb9e1ec64763d"} Nov 03 08:39:27.299278 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.299267 5015 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:27.300052 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.300031 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:27.300123 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.300063 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:27.300123 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.300073 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:27.300216 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:27.300195 5015 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-14.us-west-1.compute.internal\" not found" node="ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.346242 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.346220 5015 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:39:27.346373 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.346359 5015 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 08:39:27.349579 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.349560 5015 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:39:27.361698 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.361679 5015 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.392287 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.392270 5015 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:27.393327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.393305 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:27.393421 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.393335 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:27.393421 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.393347 5015 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-14.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:27.393421 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.393399 5015 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.400037 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.400016 5015 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:39:27.400390 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.400373 5015 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:39:27.440667 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.440646 5015 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 08:39:27.440763 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:27.440695 5015 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.440763 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.440708 5015 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.446723 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.446701 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.446804 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.446733 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6652f123a319d675474cd7afd732c542-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"6652f123a319d675474cd7afd732c542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.446804 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.446788 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.511354 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.511328 5015 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.511491 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.511404 5015 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.527259 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.527239 5015 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 08:39:27.527378 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:27.527280 5015 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.547445 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.547381 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.547445 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.547409 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6652f123a319d675474cd7afd732c542-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"6652f123a319d675474cd7afd732c542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.547445 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.547428 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.547632 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.547464 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.547632 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.547521 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6652f123a319d675474cd7afd732c542-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"6652f123a319d675474cd7afd732c542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:27.547632 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:27.547521 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b188283236722665eb092553b8ae03f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" (UID: \"0b188283236722665eb092553b8ae03f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:28.136967 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.136938 5015 apiserver.go:52] "Watching apiserver" Nov 03 08:39:28.259049 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.259022 5015 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:28.259174 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.259140 5015 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:28.292922 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.292897 5015 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 08:39:28.293691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.293664 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xs65g","openshift-network-operator/iptables-alerter-zk7xq","kube-system/kas-connection-checker-685895c7c9-5qm4n","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r","openshift-cluster-node-tuning-operator/tuned-9qzgm","openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h","openshift-ingress/router-default-6b64c9d49-xhzz8","openshift-image-registry/image-registry-64c88f488c-xq5g2","openshift-image-registry/image-registry-6d6db59d9d-pk78q","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt","openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7","openshift-dns/dns-default-6rpvp","openshift-network-console/networking-console-plugin-86c478bb88-vrpjd","kube-system/global-pull-secret-syncer-c9fnb","kube-system/kas-connection-checker-685895c7c9-p5ncs","openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx","openshift-insights/insights-operator-7c4986d449-b4ggb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z","openshift-dns/node-resolver-2xrz9","openshift-image-registry/node-ca-wkb5w","openshift-insights/insights-runtime-extractor-pc6m2","openshift-ovn-kubernetes/ovnkube-node-fnh7l","kube-system/konnectivity-agent-hp76w","openshift-multus/network-metrics-daemon-q7vbp","openshift-network-diagnostics/network-check-target-4cbq5","kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal","openshift-insights/periodic-gathering-llp7m-mp4tf","openshift-multus/multus-8hwgr","openshift-console-operator/console-operator-7cd56b8dbc-b2jvc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg","kube-system/kas-connection-checker-685895c7c9-27vdv","openshift-multus/multus-additional-cni-plugins-4chsq","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2","openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5"] Nov 03 08:39:28.295150 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.295127 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:28.295874 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.295856 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:28.296693 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.296676 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:28.297455 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.297427 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.299056 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.299039 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:28.306337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.306314 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:28.308542 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.308525 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.309540 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.309527 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:28.310487 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.310462 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.324039 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.324020 5015 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 08:39:28.324108 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.324065 5015 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:28.325637 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.325618 5015 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 08:39:28.325736 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.325658 5015 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.us-west-1.compute.internal" Nov 03 08:39:28.349320 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.349297 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.349740 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.349725 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.349820 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.349771 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.350010 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.349994 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-r6qnc\"" Nov 03 08:39:28.363103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.363082 5015 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 08:39:28.451495 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.451393 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdt5w\" (UniqueName: \"kubernetes.io/projected/e0106338-5b18-4bbf-abfe-f2eee3213c8c-kube-api-access-wdt5w\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:28.451495 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.451424 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1330da31-fcb6-4c7b-9db7-0d4d96262234-agent-certs\") pod \"konnectivity-agent-hp76w\" (UID: \"1330da31-fcb6-4c7b-9db7-0d4d96262234\") " pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:39:28.451495 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.451453 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19c1fbeb-055b-4c85-aebe-d4856252042a-crio-socket\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.451726 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.451503 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-system-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.451726 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.451533 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-hostroot\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.451726 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.451594 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-config\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.451870 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.451737 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rc2j\" (UniqueName: \"kubernetes.io/projected/d3a0d3db-7b81-4e10-b240-b665c083bda2-kube-api-access-7rc2j\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:39:28.451870 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.451837 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df61dbd1-26b9-4d1b-847c-785687d935ab-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:28.451988 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.451965 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 08:39:28.452054 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.451987 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.452106 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452073 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.452106 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452088 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.452224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452079 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 08:39:28.452224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452198 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.452320 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452249 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.452320 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452275 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-bin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.452418 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452292 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 08:39:28.452418 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452332 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.452418 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452391 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-tuned\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.452634 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452614 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.452691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452610 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-tuned\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.452691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452656 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmjl\" (UniqueName: \"kubernetes.io/projected/248b4530-5359-4e26-8309-78d1a7128494-kube-api-access-vtmjl\") pod \"kas-connection-checker-685895c7c9-27vdv\" (UID: \"248b4530-5359-4e26-8309-78d1a7128494\") " pod="kube-system/kas-connection-checker-685895c7c9-27vdv" Nov 03 08:39:28.452691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452074 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.452853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452835 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-ovn\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.452913 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452870 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:28.452986 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.452964 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0106338-5b18-4bbf-abfe-f2eee3213c8c-archives-path\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:28.453051 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453004 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-image-registry-private-configuration\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.453051 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453026 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.453051 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453037 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19c1fbeb-055b-4c85-aebe-d4856252042a-data-volume\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.453203 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453078 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-socket-dir-parent\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.453203 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453087 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0106338-5b18-4bbf-abfe-f2eee3213c8c-archives-path\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:28.453203 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453109 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.453203 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453107 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 08:39:28.453203 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453114 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19c1fbeb-055b-4c85-aebe-d4856252042a-data-volume\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.453203 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453121 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 08:39:28.453546 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453197 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpng7\" (UniqueName: \"kubernetes.io/projected/204918bd-5b87-4358-b80c-cd6262afb3b0-kube-api-access-tpng7\") pod \"network-check-source-6f78d59f99-kvwlg\" (UID: \"204918bd-5b87-4358-b80c-cd6262afb3b0\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg" Nov 03 08:39:28.453546 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453309 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-etc-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.453546 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453354 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.453546 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453400 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57491634-c53c-4571-854c-ece83d3f09a2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:28.453546 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453414 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 08:39:28.453546 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453429 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th46k\" (UniqueName: \"kubernetes.io/projected/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-kube-api-access-th46k\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:39:28.453546 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453518 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 08:39:28.453546 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453526 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysconfig\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.453546 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453513 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453553 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-lib-modules\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453583 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-tmp\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453618 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7zqh\" (UniqueName: \"kubernetes.io/projected/a21ac76c-2765-417e-832a-32e140173e03-kube-api-access-x7zqh\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453645 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453671 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-tmp-dir\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453681 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83c6ab8c-6c62-4163-a789-1edb75af2ad9-tmp\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453697 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-service-ca-bundle\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453749 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453774 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-tmp-dir\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453783 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbfl\" (UniqueName: \"kubernetes.io/projected/31ed67dd-1988-47af-a5a6-e00ee72236a3-kube-api-access-trbfl\") pod \"volume-data-source-validator-58c9f5f46d-b7vq2\" (UID: \"31ed67dd-1988-47af-a5a6-e00ee72236a3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453819 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-trusted-ca\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453846 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-trusted-ca\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453872 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-systemd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453899 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-var-lib-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.453952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453931 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.453966 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-k8s-cni-cncf-io\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454016 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-etc-kubernetes\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454068 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454114 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-env-overrides\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454144 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-var-lib-kubelet\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454218 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454260 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454297 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454300 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-default-certificate\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454307 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-env-overrides\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454344 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454392 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtbr\" (UniqueName: \"kubernetes.io/projected/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-kube-api-access-mrtbr\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454492 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74hn\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-kube-api-access-p74hn\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454547 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-netns\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.454575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454581 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8nz\" (UniqueName: \"kubernetes.io/projected/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-api-access-bm8nz\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454609 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df61dbd1-26b9-4d1b-847c-785687d935ab-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454618 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454642 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-daemon-config\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454689 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454693 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lngsh\"" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454736 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xccvh\" (UniqueName: \"kubernetes.io/projected/07f9e882-e987-44b0-a688-45e3c9abec5c-kube-api-access-xccvh\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454769 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhpmt\" (UniqueName: \"kubernetes.io/projected/bfbee40d-3926-4eae-babe-e598bfae8098-kube-api-access-bhpmt\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454791 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jv5wj\"" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454799 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-systemd\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454784 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gc7sr\"" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454825 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-bound-sa-token\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454804 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454863 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7tj\" (UniqueName: \"kubernetes.io/projected/3a650ee8-e4e1-4c94-a7da-836ea74de483-kube-api-access-ll7tj\") pod \"migrator-66df5d496b-ghcc7\" (UID: \"3a650ee8-e4e1-4c94-a7da-836ea74de483\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454896 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-installation-pull-secrets\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454903 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454866 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454927 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/283e4bdb-5662-4b18-8775-7c86dd4323f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454953 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-cni-binary-copy\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.454984 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-kubelet-config\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455016 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-service-ca-bundle\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455051 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-kubelet\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455135 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sxt9c\"" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455211 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 08:39:28.455267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455223 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-daemon-config\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455288 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455287 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455355 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455450 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455491 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v9lhl\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455616 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455614 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2st7v\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455647 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455669 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-trusted-ca\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455726 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-netns\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455754 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-device-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455779 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/587a6bc5-67cb-469b-b4b9-b35da4fb382b-config-volume\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455859 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455927 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455933 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39df63cd-3b12-4930-8a23-04ed67d961a8-cni-binary-copy\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455957 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpm6\" (UniqueName: \"kubernetes.io/projected/61655665-1941-4f5f-bc6b-41e51b2b3774-kube-api-access-4mpm6\") pod \"kas-connection-checker-685895c7c9-5qm4n\" (UID: \"61655665-1941-4f5f-bc6b-41e51b2b3774\") " pod="kube-system/kas-connection-checker-685895c7c9-5qm4n" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455976 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455995 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g5h9g\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.455992 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456001 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3a0d3db-7b81-4e10-b240-b665c083bda2-host\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456017 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456021 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456041 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-node-log\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456061 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456365 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovn-node-metrics-cert\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456408 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456495 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.456522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456412 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcv8t\" (UniqueName: \"kubernetes.io/projected/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-kube-api-access-tcv8t\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456561 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpgs\" (UniqueName: \"kubernetes.io/projected/1bedc7c0-f94a-446e-8029-c3e82609f30f-kube-api-access-6xpgs\") pod \"ingress-canary-xs65g\" (UID: \"1bedc7c0-f94a-446e-8029-c3e82609f30f\") " pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456681 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456937 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456733 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457022 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.456937 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457117 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-slash\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457247 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457288 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-sys-fs\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457378 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457375 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f27l\" (UniqueName: \"kubernetes.io/projected/bfcdc2e6-1f23-4da9-a478-0ad2175c5def-kube-api-access-6f27l\") pod \"migrator-66df5d496b-ldh5h\" (UID: \"bfcdc2e6-1f23-4da9-a478-0ad2175c5def\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457516 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-snapshots\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457560 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457594 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cnibin\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457626 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457710 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rnxpp\"" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457749 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 08:39:28.457853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457764 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-snapshots\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457923 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hvj2z\"" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457814 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-multus\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.457973 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458005 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-socket-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458035 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-hosts-file\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458077 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-sys\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458195 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m764p\" (UniqueName: \"kubernetes.io/projected/c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a-kube-api-access-m764p\") pod \"kas-connection-checker-685895c7c9-p5ncs\" (UID: \"c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a\") " pod="kube-system/kas-connection-checker-685895c7c9-p5ncs" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458253 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458267 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cwzpz\"" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458258 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458388 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jjcxd\"" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458521 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j55t5\" (UniqueName: \"kubernetes.io/projected/1959beea-2daf-4e4a-a412-9bc2d35fc277-kube-api-access-j55t5\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458551 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zgn57\"" Nov 03 08:39:28.458691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458657 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 08:39:28.459401 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458821 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 08:39:28.459401 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.458863 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-system-cni-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.459401 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459163 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 08:39:28.459401 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459168 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-stats-auth\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:28.459401 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459211 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-htlrf\"" Nov 03 08:39:28.459401 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459329 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 08:39:28.459401 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459369 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:28.459745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459542 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-config\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:28.459745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459560 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.459745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459621 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 08:39:28.459745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459641 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.459745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459721 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-stats-auth\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:28.459968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459849 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-log-socket\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.459968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459938 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.460063 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.459989 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.460063 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460022 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-config\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:28.460063 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460058 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.460224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460096 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-tmp\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.460224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460103 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 08:39:28.460224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460131 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22edddd5-3b05-4942-910a-651d537867c8-serving-cert\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.460224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460152 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 08:39:28.460398 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460278 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-crw87\"" Nov 03 08:39:28.460398 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460327 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-multus-certs\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.460524 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460431 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tt7x7\"" Nov 03 08:39:28.460524 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460469 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 08:39:28.460645 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460534 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22edddd5-3b05-4942-910a-651d537867c8-serving-cert\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.460645 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460594 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rdf8f\"" Nov 03 08:39:28.460645 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460639 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22edddd5-3b05-4942-910a-651d537867c8-tmp\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.460790 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460686 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-ca-trust-extracted\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.460790 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460699 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rk9xz\"" Nov 03 08:39:28.460790 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460713 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0e56142-8762-447d-af25-d0ba1b09b530-ca-trust-extracted\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.460790 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460721 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.461009 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460839 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.461009 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460848 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2fcj\" (UniqueName: \"kubernetes.io/projected/83c6ab8c-6c62-4163-a789-1edb75af2ad9-kube-api-access-h2fcj\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.461009 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460861 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tps59\"" Nov 03 08:39:28.461009 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460866 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-ca-trust-extracted\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.461009 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460902 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v85s\" (UniqueName: \"kubernetes.io/projected/39df63cd-3b12-4930-8a23-04ed67d961a8-kube-api-access-8v85s\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.461009 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.460978 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 08:39:28.461316 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461017 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 08:39:28.461316 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461061 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.461316 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461067 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0e56142-8762-447d-af25-d0ba1b09b530-ca-trust-extracted\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.461316 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461109 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-registration-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.461316 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461158 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.461316 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461162 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-kubernetes\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.461316 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461197 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:28.461716 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461425 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 08:39:28.461716 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461428 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vvc8d\"" Nov 03 08:39:28.461716 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461497 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46trz\" (UniqueName: \"kubernetes.io/projected/df61dbd1-26b9-4d1b-847c-785687d935ab-kube-api-access-46trz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:28.461716 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461498 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jnk2x\"" Nov 03 08:39:28.461716 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461544 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-systemd-units\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.461716 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461661 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-trusted-ca-bundle\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:28.461976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461731 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-original-pull-secret\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:28.461976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461771 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.461976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461866 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.461976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461864 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-host\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.461976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.461955 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8js\" (UniqueName: \"kubernetes.io/projected/16cb59f2-9d9f-4904-9fc8-bed9111deecd-kube-api-access-np8js\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:28.462140 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462002 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.462361 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462341 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrgh\" (UniqueName: \"kubernetes.io/projected/57491634-c53c-4571-854c-ece83d3f09a2-kube-api-access-jmrgh\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:28.462451 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462385 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:28.462451 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462419 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.462451 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462418 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdq7\" (UniqueName: \"kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7\") pod \"network-check-target-4cbq5\" (UID: \"f70226b0-5f4c-465c-9ec7-7ceee1bb376f\") " pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:28.462451 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462441 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1330da31-fcb6-4c7b-9db7-0d4d96262234-agent-certs\") pod \"konnectivity-agent-hp76w\" (UID: \"1330da31-fcb6-4c7b-9db7-0d4d96262234\") " pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:39:28.462730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462517 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 08:39:28.462730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462527 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-config\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.462730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462617 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a21ac76c-2765-417e-832a-32e140173e03-serving-cert\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:28.462730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462619 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.462730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462644 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-cnibin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.462730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462617 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.462730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462663 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-kubelet\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.462730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462673 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:28.462730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462692 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:28.462730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462701 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-script-lib\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.463015 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462780 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:28.463015 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462786 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-etc-selinux\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.463015 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462778 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sntnv\"" Nov 03 08:39:28.463015 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462804 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a21ac76c-2765-417e-832a-32e140173e03-serving-cert\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:28.463015 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462853 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-conf\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.463015 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462882 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5x8t\" (UniqueName: \"kubernetes.io/projected/70bfc17c-8e18-473a-a8fe-e386112cfaa4-kube-api-access-g5x8t\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:28.463015 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462908 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bedc7c0-f94a-446e-8029-c3e82609f30f-cert\") pod \"ingress-canary-xs65g\" (UID: \"1bedc7c0-f94a-446e-8029-c3e82609f30f\") " pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:28.463015 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.462976 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07f9e882-e987-44b0-a688-45e3c9abec5c-host-slash\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463052 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/587a6bc5-67cb-469b-b4b9-b35da4fb382b-tmp-dir\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463082 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gnnf\" (UniqueName: \"kubernetes.io/projected/63810bfe-e55b-4524-bde4-8f072b4b63a3-kube-api-access-4gnnf\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463111 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27j2\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-kube-api-access-z27j2\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463118 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/587a6bc5-67cb-469b-b4b9-b35da4fb382b-tmp-dir\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463138 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqkf\" (UniqueName: \"kubernetes.io/projected/587a6bc5-67cb-469b-b4b9-b35da4fb382b-kube-api-access-qlqkf\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463149 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bedc7c0-f94a-446e-8029-c3e82609f30f-cert\") pod \"ingress-canary-xs65g\" (UID: \"1bedc7c0-f94a-446e-8029-c3e82609f30f\") " pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463164 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtzp\" (UniqueName: \"kubernetes.io/projected/22edddd5-3b05-4942-910a-651d537867c8-kube-api-access-4qtzp\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463197 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-registry-certificates\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463224 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-conf-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463255 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463280 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-netd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463293 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-image-registry-private-configuration\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463303 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-modprobe-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463342 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463457 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-bound-sa-token\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.463552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463523 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07f9e882-e987-44b0-a688-45e3c9abec5c-iptables-alerter-script\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463572 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-run\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463583 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-registry-certificates\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463677 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-dbus\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463748 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16cb59f2-9d9f-4904-9fc8-bed9111deecd-serving-cert\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463757 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07f9e882-e987-44b0-a688-45e3c9abec5c-iptables-alerter-script\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463776 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/16cb59f2-9d9f-4904-9fc8-bed9111deecd-config\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463803 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-binary-copy\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463836 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463870 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-image-registry-private-configuration\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463919 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-bin\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463965 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.463978 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cni-binary-copy\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464008 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16cb59f2-9d9f-4904-9fc8-bed9111deecd-serving-cert\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464029 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1330da31-fcb6-4c7b-9db7-0d4d96262234-konnectivity-ca\") pod \"konnectivity-agent-hp76w\" (UID: \"1330da31-fcb6-4c7b-9db7-0d4d96262234\") " pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464049 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-service-ca-bundle\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464062 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3a0d3db-7b81-4e10-b240-b665c083bda2-serviceca\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464084 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/16cb59f2-9d9f-4904-9fc8-bed9111deecd-config\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464099 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-certificates\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464139 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-image-registry-private-configuration\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464190 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1330da31-fcb6-4c7b-9db7-0d4d96262234-konnectivity-ca\") pod \"konnectivity-agent-hp76w\" (UID: \"1330da31-fcb6-4c7b-9db7-0d4d96262234\") " pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:39:28.464226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464210 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-installation-pull-secrets\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.464864 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464260 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3a0d3db-7b81-4e10-b240-b665c083bda2-serviceca\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:39:28.464864 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464308 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-certificates\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.464864 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464853 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-default-certificate\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:28.464949 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.464873 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df61dbd1-26b9-4d1b-847c-785687d935ab-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:28.466034 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.466019 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-service-ca-bundle\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:28.466099 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.466069 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/283e4bdb-5662-4b18-8775-7c86dd4323f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:28.466205 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.466188 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/587a6bc5-67cb-469b-b4b9-b35da4fb382b-config-volume\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:28.467092 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.467076 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovn-node-metrics-cert\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.472382 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.472368 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df61dbd1-26b9-4d1b-847c-785687d935ab-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:28.497632 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.497611 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hdxz9\"" Nov 03 08:39:28.499972 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.499954 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jw5dx\"" Nov 03 08:39:28.502183 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.502166 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8lmql\"" Nov 03 08:39:28.502927 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.502910 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 08:39:28.503107 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.502915 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 08:39:28.503275 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.503169 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1959beea-2daf-4e4a-a412-9bc2d35fc277-ovnkube-script-lib\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.503435 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.503419 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 08:39:28.503819 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.503786 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57491634-c53c-4571-854c-ece83d3f09a2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:28.504634 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.504612 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-installation-pull-secrets\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.506083 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.506053 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-installation-pull-secrets\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.506181 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.506115 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 08:39:28.506181 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.506121 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 08:39:28.506333 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.506313 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 08:39:28.509153 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.509136 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22edddd5-3b05-4942-910a-651d537867c8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.509921 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.509904 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdt5w\" (UniqueName: \"kubernetes.io/projected/e0106338-5b18-4bbf-abfe-f2eee3213c8c-kube-api-access-wdt5w\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:28.512753 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.512737 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-trusted-ca-bundle\") pod \"periodic-gathering-llp7m-mp4tf\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:39:28.514453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.514434 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a21ac76c-2765-417e-832a-32e140173e03-trusted-ca\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:28.515001 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.514982 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-trusted-ca\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.516023 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.515998 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmjl\" (UniqueName: \"kubernetes.io/projected/248b4530-5359-4e26-8309-78d1a7128494-kube-api-access-vtmjl\") pod \"kas-connection-checker-685895c7c9-27vdv\" (UID: \"248b4530-5359-4e26-8309-78d1a7128494\") " pod="kube-system/kas-connection-checker-685895c7c9-27vdv" Nov 03 08:39:28.516127 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.516031 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rc2j\" (UniqueName: \"kubernetes.io/projected/d3a0d3db-7b81-4e10-b240-b665c083bda2-kube-api-access-7rc2j\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:39:28.516259 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.516129 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtbr\" (UniqueName: \"kubernetes.io/projected/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-kube-api-access-mrtbr\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:28.516259 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.516149 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th46k\" (UniqueName: \"kubernetes.io/projected/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-kube-api-access-th46k\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:39:28.516259 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.516220 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7zqh\" (UniqueName: \"kubernetes.io/projected/a21ac76c-2765-417e-832a-32e140173e03-kube-api-access-x7zqh\") pod \"console-operator-7cd56b8dbc-b2jvc\" (UID: \"a21ac76c-2765-417e-832a-32e140173e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:28.516259 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.516222 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbfl\" (UniqueName: \"kubernetes.io/projected/31ed67dd-1988-47af-a5a6-e00ee72236a3-kube-api-access-trbfl\") pod \"volume-data-source-validator-58c9f5f46d-b7vq2\" (UID: \"31ed67dd-1988-47af-a5a6-e00ee72236a3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b7vq2" Nov 03 08:39:28.516554 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.516310 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpng7\" (UniqueName: \"kubernetes.io/projected/204918bd-5b87-4358-b80c-cd6262afb3b0-kube-api-access-tpng7\") pod \"network-check-source-6f78d59f99-kvwlg\" (UID: \"204918bd-5b87-4358-b80c-cd6262afb3b0\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kvwlg" Nov 03 08:39:28.516554 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.516363 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-trusted-ca\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.516554 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.516463 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-bound-sa-token\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.516810 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.516792 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7tj\" (UniqueName: \"kubernetes.io/projected/3a650ee8-e4e1-4c94-a7da-836ea74de483-kube-api-access-ll7tj\") pod \"migrator-66df5d496b-ghcc7\" (UID: \"3a650ee8-e4e1-4c94-a7da-836ea74de483\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7" Nov 03 08:39:28.516882 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.516860 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhpmt\" (UniqueName: \"kubernetes.io/projected/bfbee40d-3926-4eae-babe-e598bfae8098-kube-api-access-bhpmt\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.516993 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.516980 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xccvh\" (UniqueName: \"kubernetes.io/projected/07f9e882-e987-44b0-a688-45e3c9abec5c-kube-api-access-xccvh\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:39:28.517672 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.517644 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcv8t\" (UniqueName: \"kubernetes.io/projected/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-kube-api-access-tcv8t\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.517762 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.517748 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74hn\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-kube-api-access-p74hn\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.518192 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.518173 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m764p\" (UniqueName: \"kubernetes.io/projected/c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a-kube-api-access-m764p\") pod \"kas-connection-checker-685895c7c9-p5ncs\" (UID: \"c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a\") " pod="kube-system/kas-connection-checker-685895c7c9-p5ncs" Nov 03 08:39:28.518586 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.518568 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpm6\" (UniqueName: \"kubernetes.io/projected/61655665-1941-4f5f-bc6b-41e51b2b3774-kube-api-access-4mpm6\") pod \"kas-connection-checker-685895c7c9-5qm4n\" (UID: \"61655665-1941-4f5f-bc6b-41e51b2b3774\") " pod="kube-system/kas-connection-checker-685895c7c9-5qm4n" Nov 03 08:39:28.519167 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.519143 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8nz\" (UniqueName: \"kubernetes.io/projected/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-api-access-bm8nz\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.519275 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.519194 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f27l\" (UniqueName: \"kubernetes.io/projected/bfcdc2e6-1f23-4da9-a478-0ad2175c5def-kube-api-access-6f27l\") pod \"migrator-66df5d496b-ldh5h\" (UID: \"bfcdc2e6-1f23-4da9-a478-0ad2175c5def\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h" Nov 03 08:39:28.519345 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.519290 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j55t5\" (UniqueName: \"kubernetes.io/projected/1959beea-2daf-4e4a-a412-9bc2d35fc277-kube-api-access-j55t5\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.519401 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.519385 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v85s\" (UniqueName: \"kubernetes.io/projected/39df63cd-3b12-4930-8a23-04ed67d961a8-kube-api-access-8v85s\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.519497 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.519456 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2fcj\" (UniqueName: \"kubernetes.io/projected/83c6ab8c-6c62-4163-a789-1edb75af2ad9-kube-api-access-h2fcj\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.519611 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.519594 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpgs\" (UniqueName: \"kubernetes.io/projected/1bedc7c0-f94a-446e-8029-c3e82609f30f-kube-api-access-6xpgs\") pod \"ingress-canary-xs65g\" (UID: \"1bedc7c0-f94a-446e-8029-c3e82609f30f\") " pod="openshift-ingress-canary/ingress-canary-xs65g" Nov 03 08:39:28.520623 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.520601 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrgh\" (UniqueName: \"kubernetes.io/projected/57491634-c53c-4571-854c-ece83d3f09a2-kube-api-access-jmrgh\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:28.520729 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.520700 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46trz\" (UniqueName: \"kubernetes.io/projected/df61dbd1-26b9-4d1b-847c-785687d935ab-kube-api-access-46trz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ck58r\" (UID: \"df61dbd1-26b9-4d1b-847c-785687d935ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" Nov 03 08:39:28.521220 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.521200 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqkf\" (UniqueName: \"kubernetes.io/projected/587a6bc5-67cb-469b-b4b9-b35da4fb382b-kube-api-access-qlqkf\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:28.521501 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.521465 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qdq7\" (UniqueName: \"kubernetes.io/projected/f70226b0-5f4c-465c-9ec7-7ceee1bb376f-kube-api-access-4qdq7\") pod \"network-check-target-4cbq5\" (UID: \"f70226b0-5f4c-465c-9ec7-7ceee1bb376f\") " pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:28.521646 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.521630 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gnnf\" (UniqueName: \"kubernetes.io/projected/63810bfe-e55b-4524-bde4-8f072b4b63a3-kube-api-access-4gnnf\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:28.521691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.521672 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-bound-sa-token\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.521822 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.521805 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5x8t\" (UniqueName: \"kubernetes.io/projected/70bfc17c-8e18-473a-a8fe-e386112cfaa4-kube-api-access-g5x8t\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:28.522658 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.522635 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8js\" (UniqueName: \"kubernetes.io/projected/16cb59f2-9d9f-4904-9fc8-bed9111deecd-kube-api-access-np8js\") pod \"service-ca-operator-544d749c8c-s5kxx\" (UID: \"16cb59f2-9d9f-4904-9fc8-bed9111deecd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" Nov 03 08:39:28.522824 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.522806 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtzp\" (UniqueName: \"kubernetes.io/projected/22edddd5-3b05-4942-910a-651d537867c8-kube-api-access-4qtzp\") pod \"insights-operator-7c4986d449-b4ggb\" (UID: \"22edddd5-3b05-4942-910a-651d537867c8\") " pod="openshift-insights/insights-operator-7c4986d449-b4ggb" Nov 03 08:39:28.522879 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.522868 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27j2\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-kube-api-access-z27j2\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.565051 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565029 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-var-lib-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.565051 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565055 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.565170 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565070 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-k8s-cni-cncf-io\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.565170 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565085 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-etc-kubernetes\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.565170 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565138 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-k8s-cni-cncf-io\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.565170 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565142 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-var-lib-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.565170 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565156 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.565313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565179 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-etc-kubernetes\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.565313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565203 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-var-lib-kubelet\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.565313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565214 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.565313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565227 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-netns\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.565313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565208 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.565313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565246 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-systemd\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.565313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565261 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-var-lib-kubelet\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.565313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565259 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-netns\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.565313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565278 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-kubelet-config\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:28.565313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565305 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-kubelet\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.565313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565314 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-kubelet-config\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565325 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-systemd\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565328 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-netns\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565350 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-kubelet\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565364 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-device-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565382 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3a0d3db-7b81-4e10-b240-b665c083bda2-host\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565401 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-node-log\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565415 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-netns\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565430 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-device-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565426 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565458 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-node-log\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565464 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3a0d3db-7b81-4e10-b240-b665c083bda2-host\") pod \"node-ca-wkb5w\" (UID: \"d3a0d3db-7b81-4e10-b240-b665c083bda2\") " pod="openshift-image-registry/node-ca-wkb5w" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565494 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-slash\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565529 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-slash\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565535 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-sys-fs\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.565564 5015 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565584 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cnibin\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565605 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565638 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-sys-fs\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.565641 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls podName:19c1fbeb-055b-4c85-aebe-d4856252042a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.065618751 +0000 UTC m=+1.990560958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pc6m2" (UID: "19c1fbeb-055b-4c85-aebe-d4856252042a") : secret "insights-runtime-extractor-tls" not found Nov 03 08:39:28.565651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565655 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-cnibin\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565665 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-multus\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565683 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.565704 5015 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565734 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-socket-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565737 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-multus\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565703 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-socket-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.565768 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs podName:63810bfe-e55b-4524-bde4-8f072b4b63a3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.065755109 +0000 UTC m=+1.990697303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs") pod "network-metrics-daemon-q7vbp" (UID: "63810bfe-e55b-4524-bde4-8f072b4b63a3") : secret "metrics-daemon-secret" not found Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565777 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565785 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-hosts-file\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565821 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-sys\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565812 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70d5c7ae-f8aa-4300-b5a6-c76273b66b3c-hosts-file\") pod \"node-resolver-2xrz9\" (UID: \"70d5c7ae-f8aa-4300-b5a6-c76273b66b3c\") " pod="openshift-dns/node-resolver-2xrz9" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565847 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565872 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-system-cni-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565915 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-system-cni-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565916 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-sys\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.565960 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.065944982 +0000 UTC m=+1.990887388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : configmap references non-existent config key: service-ca.crt Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.565991 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566015 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566092 5015 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566104 5015 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6db59d9d-pk78q: secret "image-registry-tls" not found Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566117 5015 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566142 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-log-socket\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566117 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-log-socket\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566171 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls podName:a0e56142-8762-447d-af25-d0ba1b09b530 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.066157789 +0000 UTC m=+1.991100014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls") pod "image-registry-6d6db59d9d-pk78q" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530") : secret "image-registry-tls" not found Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566194 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.066187519 +0000 UTC m=+1.991129714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : secret "router-metrics-certs-default" not found Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566214 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-multus-certs\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566232 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-registration-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566254 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-kubernetes\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566275 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-systemd-units\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.566302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566291 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566325 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-registration-dir\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566336 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-run-multus-certs\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566351 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-kubernetes\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566361 5015 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566372 5015 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c88f488c-xq5g2: secret "image-registry-tls" not found Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566363 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-systemd-units\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566370 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-host\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566405 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls podName:1b1845f9-4e7a-40b2-922a-95e1cb82eeea nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.066394069 +0000 UTC m=+1.991336298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls") pod "image-registry-64c88f488c-xq5g2" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea") : secret "image-registry-tls" not found Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566427 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-cnibin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566441 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-kubelet\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566456 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-etc-selinux\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566492 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-conf\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566501 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-cnibin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566406 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-host\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566519 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07f9e882-e987-44b0-a688-45e3c9abec5c-host-slash\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566541 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-kubelet\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566549 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfbee40d-3926-4eae-babe-e598bfae8098-etc-selinux\") pod \"aws-ebs-csi-driver-node-82g5z\" (UID: \"bfbee40d-3926-4eae-babe-e598bfae8098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82g5z" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566566 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-conf-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566583 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566603 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-netd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566614 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysctl-conf\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566626 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-modprobe-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566644 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566650 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-conf-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566654 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566671 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-netd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566617 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07f9e882-e987-44b0-a688-45e3c9abec5c-host-slash\") pod \"iptables-alerter-zk7xq\" (UID: \"07f9e882-e987-44b0-a688-45e3c9abec5c\") " pod="openshift-network-operator/iptables-alerter-zk7xq" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566691 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-run\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566717 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-dbus\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566721 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-modprobe-d\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566724 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-run\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566757 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25b42294-fcbd-4bc8-8652-9d4d7e762e3d-dbus\") pod \"global-pull-secret-syncer-c9fnb\" (UID: \"25b42294-fcbd-4bc8-8652-9d4d7e762e3d\") " pod="kube-system/global-pull-secret-syncer-c9fnb" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566775 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566799 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-bin\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566822 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566863 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-cni-bin\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566881 5015 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566894 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19c1fbeb-055b-4c85-aebe-d4856252042a-crio-socket\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566914 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert podName:283e4bdb-5662-4b18-8775-7c86dd4323f3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.066905212 +0000 UTC m=+1.991847432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vrpjd" (UID: "283e4bdb-5662-4b18-8775-7c86dd4323f3") : secret "networking-console-plugin-cert" not found Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566922 5015 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566927 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-system-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566934 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19c1fbeb-055b-4c85-aebe-d4856252042a-crio-socket\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566953 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-hostroot\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.566962 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls podName:57491634-c53c-4571-854c-ece83d3f09a2 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.066949824 +0000 UTC m=+1.991892039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jrqm5" (UID: "57491634-c53c-4571-854c-ece83d3f09a2") : secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566973 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-hostroot\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566983 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-system-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.566999 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-bin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567016 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-host-var-lib-cni-bin\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567029 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567055 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-ovn\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567077 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0893a4fe-c6b7-42e5-8e5c-8d179dfefedc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4chsq\" (UID: \"0893a4fe-c6b7-42e5-8e5c-8d179dfefedc\") " pod="openshift-multus/multus-additional-cni-plugins-4chsq" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567079 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567111 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-socket-dir-parent\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567119 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-ovn\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567127 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-etc-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.567138 5015 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567146 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-etc-openvswitch\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567158 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.567183 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls podName:587a6bc5-67cb-469b-b4b9-b35da4fb382b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.067170308 +0000 UTC m=+1.992112524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls") pod "dns-default-6rpvp" (UID: "587a6bc5-67cb-469b-b4b9-b35da4fb382b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567196 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-host-run-ovn-kubernetes\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567202 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19c1fbeb-055b-4c85-aebe-d4856252042a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567209 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysconfig\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567197 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-socket-dir-parent\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567233 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-lib-modules\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567257 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567272 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-etc-sysconfig\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567283 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567315 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83c6ab8c-6c62-4163-a789-1edb75af2ad9-lib-modules\") pod \"tuned-9qzgm\" (UID: \"83c6ab8c-6c62-4163-a789-1edb75af2ad9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qzgm" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567318 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-systemd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567335 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39df63cd-3b12-4930-8a23-04ed67d961a8-multus-cni-dir\") pod \"multus-8hwgr\" (UID: \"39df63cd-3b12-4930-8a23-04ed67d961a8\") " pod="openshift-multus/multus-8hwgr" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.567353 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1959beea-2daf-4e4a-a412-9bc2d35fc277-run-systemd\") pod \"ovnkube-node-fnh7l\" (UID: \"1959beea-2daf-4e4a-a412-9bc2d35fc277\") " pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:28.567337 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.567326 5015 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 08:39:28.569265 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:28.567433 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls podName:7bebbc2e-c649-4f7f-bef3-0a8f158d6cae nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.067417428 +0000 UTC m=+1.992359627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgcwt" (UID: "7bebbc2e-c649-4f7f-bef3-0a8f158d6cae") : secret "samples-operator-tls" not found Nov 03 08:39:28.597359 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.597337 5015 scope.go:122] "RemoveContainer" containerID="98480b8ec484ff22c4640ad44514431fb0b225906819ec8a69dfdc4b153fff24" Nov 03 08:39:28.612510 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:28.612491 5015 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:39:29.072281 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.072201 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:29.072281 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.072259 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.072292 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.072331 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072360 5015 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.072366 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072379 5015 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c88f488c-xq5g2: secret "image-registry-tls" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.072423 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072435 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls podName:1b1845f9-4e7a-40b2-922a-95e1cb82eeea nodeName:}" failed. No retries permitted until 2026-11-03 08:39:30.072417559 +0000 UTC m=+2.997359768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls") pod "image-registry-64c88f488c-xq5g2" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea") : secret "image-registry-tls" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072442 5015 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072442 5015 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072507 5015 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072520 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls podName:57491634-c53c-4571-854c-ece83d3f09a2 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:30.07250315 +0000 UTC m=+2.997445345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jrqm5" (UID: "57491634-c53c-4571-854c-ece83d3f09a2") : secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072543 5015 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072515 5015 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072564 5015 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.072468 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072561 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert podName:283e4bdb-5662-4b18-8775-7c86dd4323f3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:30.072544803 +0000 UTC m=+2.997487012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vrpjd" (UID: "283e4bdb-5662-4b18-8775-7c86dd4323f3") : secret "networking-console-plugin-cert" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072619 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls podName:587a6bc5-67cb-469b-b4b9-b35da4fb382b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:30.072603046 +0000 UTC m=+2.997545241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls") pod "dns-default-6rpvp" (UID: "587a6bc5-67cb-469b-b4b9-b35da4fb382b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072639 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs podName:63810bfe-e55b-4524-bde4-8f072b4b63a3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:30.072630344 +0000 UTC m=+2.997572546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs") pod "network-metrics-daemon-q7vbp" (UID: "63810bfe-e55b-4524-bde4-8f072b4b63a3") : secret "metrics-daemon-secret" not found Nov 03 08:39:29.072661 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072664 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls podName:19c1fbeb-055b-4c85-aebe-d4856252042a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:30.07265088 +0000 UTC m=+2.997593075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pc6m2" (UID: "19c1fbeb-055b-4c85-aebe-d4856252042a") : secret "insights-runtime-extractor-tls" not found Nov 03 08:39:29.073326 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072686 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls podName:7bebbc2e-c649-4f7f-bef3-0a8f158d6cae nodeName:}" failed. No retries permitted until 2026-11-03 08:39:30.072677842 +0000 UTC m=+2.997620038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgcwt" (UID: "7bebbc2e-c649-4f7f-bef3-0a8f158d6cae") : secret "samples-operator-tls" not found Nov 03 08:39:29.073326 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.072730 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:29.073326 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.072764 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:29.073326 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.072792 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:29.073326 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072855 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:30.072843354 +0000 UTC m=+2.997785572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : configmap references non-existent config key: service-ca.crt Nov 03 08:39:29.073326 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072857 5015 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:29.073326 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072872 5015 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6db59d9d-pk78q: secret "image-registry-tls" not found Nov 03 08:39:29.073326 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072880 5015 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 08:39:29.073326 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072902 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls podName:a0e56142-8762-447d-af25-d0ba1b09b530 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:30.07289241 +0000 UTC m=+2.997834659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls") pod "image-registry-6d6db59d9d-pk78q" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530") : secret "image-registry-tls" not found Nov 03 08:39:29.073326 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.072959 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:30.072950378 +0000 UTC m=+2.997892572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : secret "router-metrics-certs-default" not found Nov 03 08:39:29.083164 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.083140 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2xrz9_70d5c7ae-f8aa-4300-b5a6-c76273b66b3c/dns-node-resolver/0.log" Nov 03 08:39:29.263822 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.263795 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/2.log" Nov 03 08:39:29.264223 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.264206 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/1.log" Nov 03 08:39:29.264275 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.264235 5015 generic.go:356] "Generic (PLEG): container finished" podID="a21ac76c-2765-417e-832a-32e140173e03" containerID="3a14aa4266aa2c82a4bd1d02c088d5f777afc4bc2b6d1114c870936fb1dc645b" exitCode=255 Nov 03 08:39:29.264341 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.264319 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" event={"ID":"a21ac76c-2765-417e-832a-32e140173e03","Type":"ContainerDied","Data":"3a14aa4266aa2c82a4bd1d02c088d5f777afc4bc2b6d1114c870936fb1dc645b"} Nov 03 08:39:29.264400 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.264386 5015 scope.go:122] "RemoveContainer" containerID="98480b8ec484ff22c4640ad44514431fb0b225906819ec8a69dfdc4b153fff24" Nov 03 08:39:29.264818 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:29.264795 5015 scope.go:122] "RemoveContainer" containerID="3a14aa4266aa2c82a4bd1d02c088d5f777afc4bc2b6d1114c870936fb1dc645b" Nov 03 08:39:29.264953 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:29.264939 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b2jvc_openshift-console-operator(a21ac76c-2765-417e-832a-32e140173e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podUID="a21ac76c-2765-417e-832a-32e140173e03" Nov 03 08:39:30.082176 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.082140 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:30.082176 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.082181 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.082207 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.082234 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.082274 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.082295 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082311 5015 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.082315 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082311 5015 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082373 5015 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082388 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert podName:283e4bdb-5662-4b18-8775-7c86dd4323f3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:32.082364473 +0000 UTC m=+5.007306670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-vrpjd" (UID: "283e4bdb-5662-4b18-8775-7c86dd4323f3") : secret "networking-console-plugin-cert" not found Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082396 5015 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082404 5015 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082423 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls podName:7bebbc2e-c649-4f7f-bef3-0a8f158d6cae nodeName:}" failed. No retries permitted until 2026-11-03 08:39:32.082406957 +0000 UTC m=+5.007349153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgcwt" (UID: "7bebbc2e-c649-4f7f-bef3-0a8f158d6cae") : secret "samples-operator-tls" not found Nov 03 08:39:30.082442 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082444 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:32.082432533 +0000 UTC m=+5.007374737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : configmap references non-existent config key: service-ca.crt Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082467 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls podName:19c1fbeb-055b-4c85-aebe-d4856252042a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:32.082449962 +0000 UTC m=+5.007392174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pc6m2" (UID: "19c1fbeb-055b-4c85-aebe-d4856252042a") : secret "insights-runtime-extractor-tls" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082312 5015 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.082524 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082562 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls podName:587a6bc5-67cb-469b-b4b9-b35da4fb382b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:32.082539671 +0000 UTC m=+5.007481867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls") pod "dns-default-6rpvp" (UID: "587a6bc5-67cb-469b-b4b9-b35da4fb382b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082603 5015 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082605 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs podName:63810bfe-e55b-4524-bde4-8f072b4b63a3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:32.082599208 +0000 UTC m=+5.007541403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs") pod "network-metrics-daemon-q7vbp" (UID: "63810bfe-e55b-4524-bde4-8f072b4b63a3") : secret "metrics-daemon-secret" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082621 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls podName:57491634-c53c-4571-854c-ece83d3f09a2 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:32.082616083 +0000 UTC m=+5.007558278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jrqm5" (UID: "57491634-c53c-4571-854c-ece83d3f09a2") : secret "cluster-monitoring-operator-tls" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.082635 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082642 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:32.082629952 +0000 UTC m=+5.007572166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : secret "router-metrics-certs-default" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.082686 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082691 5015 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082744 5015 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6db59d9d-pk78q: secret "image-registry-tls" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082744 5015 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082757 5015 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c88f488c-xq5g2: secret "image-registry-tls" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082788 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls podName:a0e56142-8762-447d-af25-d0ba1b09b530 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:32.082775561 +0000 UTC m=+5.007717761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls") pod "image-registry-6d6db59d9d-pk78q" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530") : secret "image-registry-tls" not found Nov 03 08:39:30.083024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:30.082804 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls podName:1b1845f9-4e7a-40b2-922a-95e1cb82eeea nodeName:}" failed. No retries permitted until 2026-11-03 08:39:32.082795533 +0000 UTC m=+5.007737738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls") pod "image-registry-64c88f488c-xq5g2" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea") : secret "image-registry-tls" not found Nov 03 08:39:30.273161 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.273116 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ghcc7" event={"ID":"3a650ee8-e4e1-4c94-a7da-836ea74de483","Type":"ContainerStarted","Data":"c3cf47653e90230ef928f1c715079027096a64b997f8b9c2cf9b9ed1d44834e3"} Nov 03 08:39:30.274629 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.274610 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/2.log" Nov 03 08:39:30.276160 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.276139 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldh5h" event={"ID":"bfcdc2e6-1f23-4da9-a478-0ad2175c5def","Type":"ContainerStarted","Data":"7e39bf1fcd910e32aa64d938c8110e4fe485f51934d791205ac2be08145efee1"} Nov 03 08:39:30.488611 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:30.488587 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wkb5w_d3a0d3db-7b81-4e10-b240-b665c083bda2/node-ca/0.log" Nov 03 08:39:31.014447 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:31.014416 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:39:31.283356 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:31.283276 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xs65g_1bedc7c0-f94a-446e-8029-c3e82609f30f/serve-healthcheck-canary/0.log" Nov 03 08:39:31.515244 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:31.515217 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ghcc7_3a650ee8-e4e1-4c94-a7da-836ea74de483/migrator/0.log" Nov 03 08:39:31.683416 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:31.683348 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ldh5h_bfcdc2e6-1f23-4da9-a478-0ad2175c5def/migrator/0.log" Nov 03 08:39:31.885230 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:31.885202 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-ck58r_df61dbd1-26b9-4d1b-847c-785687d935ab/kube-storage-version-migrator-operator/0.log" Nov 03 08:39:32.102656 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.102615 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.102679 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.102702 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.102729 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.102750 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.102763 5015 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.102774 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.102811 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.102839 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls podName:7bebbc2e-c649-4f7f-bef3-0a8f158d6cae nodeName:}" failed. No retries permitted until 2026-11-03 08:39:36.102818285 +0000 UTC m=+9.027760498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-jgcwt" (UID: "7bebbc2e-c649-4f7f-bef3-0a8f158d6cae") : secret "samples-operator-tls" not found Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.102844 5015 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.102873 5015 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.102884 5015 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c88f488c-xq5g2: secret "image-registry-tls" not found Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.102883 5015 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 08:39:32.102892 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.102895 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs podName:63810bfe-e55b-4524-bde4-8f072b4b63a3 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:36.102879668 +0000 UTC m=+9.027821883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs") pod "network-metrics-daemon-q7vbp" (UID: "63810bfe-e55b-4524-bde4-8f072b4b63a3") : secret "metrics-daemon-secret" not found Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.102920 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls podName:1b1845f9-4e7a-40b2-922a-95e1cb82eeea nodeName:}" failed. No retries permitted until 2026-11-03 08:39:36.102910654 +0000 UTC m=+9.027852849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls") pod "image-registry-64c88f488c-xq5g2" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea") : secret "image-registry-tls" not found Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.102947 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs podName:70bfc17c-8e18-473a-a8fe-e386112cfaa4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:36.102938994 +0000 UTC m=+9.027881197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs") pod "router-default-6b64c9d49-xhzz8" (UID: "70bfc17c-8e18-473a-a8fe-e386112cfaa4") : secret "router-metrics-certs-default" not found Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.102965 5015 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.102973 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.102983 5015 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6db59d9d-pk78q: secret "image-registry-tls" not found Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.102998 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.103030 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls podName:a0e56142-8762-447d-af25-d0ba1b09b530 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:36.103015655 +0000 UTC m=+9.027957852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls") pod "image-registry-6d6db59d9d-pk78q" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530") : secret "image-registry-tls" not found Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.103077 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.103131 5015 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.103195 5015 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:32.103247 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls podName:587a6bc5-67cb-469b-b4b9-b35da4fb382b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:36.103230667 +0000 UTC m=+9.028172862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls") pod "dns-default-6rpvp" (UID: "587a6bc5-67cb-469b-b4b9-b35da4fb382b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:32.103626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.103583 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70bfc17c-8e18-473a-a8fe-e386112cfaa4-service-ca-bundle\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:32.105521 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.105498 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/283e4bdb-5662-4b18-8775-7c86dd4323f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-vrpjd\" (UID: \"283e4bdb-5662-4b18-8775-7c86dd4323f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:32.105604 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.105504 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19c1fbeb-055b-4c85-aebe-d4856252042a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc6m2\" (UID: \"19c1fbeb-055b-4c85-aebe-d4856252042a\") " pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:32.105604 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.105551 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57491634-c53c-4571-854c-ece83d3f09a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jrqm5\" (UID: \"57491634-c53c-4571-854c-ece83d3f09a2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:32.195990 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.195960 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" Nov 03 08:39:32.200261 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.200235 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" Nov 03 08:39:32.211511 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.211491 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pc6m2" Nov 03 08:39:32.549651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.549626 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:32.685384 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.685351 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-vrpjd"] Nov 03 08:39:32.688856 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.688833 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5"] Nov 03 08:39:32.689285 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:32.689266 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pc6m2"] Nov 03 08:39:32.728340 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:32.728307 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283e4bdb_5662_4b18_8775_7c86dd4323f3.slice/crio-ef82670fdcefea7b94224b1a28017c0adfffb18531dd97ff7c6633615c93b9e8 WatchSource:0}: Error finding container ef82670fdcefea7b94224b1a28017c0adfffb18531dd97ff7c6633615c93b9e8: Status 404 returned error can't find the container with id ef82670fdcefea7b94224b1a28017c0adfffb18531dd97ff7c6633615c93b9e8 Nov 03 08:39:32.731421 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:32.731395 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57491634_c53c_4571_854c_ece83d3f09a2.slice/crio-49edb0f5de6455017a1311bfddc7ce3c4649cb37e71cf39ea92a6b07fb9d3586 WatchSource:0}: Error finding container 49edb0f5de6455017a1311bfddc7ce3c4649cb37e71cf39ea92a6b07fb9d3586: Status 404 returned error can't find the container with id 49edb0f5de6455017a1311bfddc7ce3c4649cb37e71cf39ea92a6b07fb9d3586 Nov 03 08:39:32.732281 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:32.732256 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c1fbeb_055b_4c85_aebe_d4856252042a.slice/crio-bc54145196a817e9f91374840b61d9b4c11074b843c62a1455a80a0480e10a5c WatchSource:0}: Error finding container bc54145196a817e9f91374840b61d9b4c11074b843c62a1455a80a0480e10a5c: Status 404 returned error can't find the container with id bc54145196a817e9f91374840b61d9b4c11074b843c62a1455a80a0480e10a5c Nov 03 08:39:33.288361 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:33.288320 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pc6m2" event={"ID":"19c1fbeb-055b-4c85-aebe-d4856252042a","Type":"ContainerStarted","Data":"ce1ca75f15c408232c8e110b9e8b068db65dfa6b10634503345581db81709595"} Nov 03 08:39:33.288569 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:33.288386 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pc6m2" event={"ID":"19c1fbeb-055b-4c85-aebe-d4856252042a","Type":"ContainerStarted","Data":"bc54145196a817e9f91374840b61d9b4c11074b843c62a1455a80a0480e10a5c"} Nov 03 08:39:33.289612 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:33.289590 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" event={"ID":"283e4bdb-5662-4b18-8775-7c86dd4323f3","Type":"ContainerStarted","Data":"ef82670fdcefea7b94224b1a28017c0adfffb18531dd97ff7c6633615c93b9e8"} Nov 03 08:39:33.290772 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:33.290750 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" event={"ID":"57491634-c53c-4571-854c-ece83d3f09a2","Type":"ContainerStarted","Data":"49edb0f5de6455017a1311bfddc7ce3c4649cb37e71cf39ea92a6b07fb9d3586"} Nov 03 08:39:34.297462 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:34.297413 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pc6m2" event={"ID":"19c1fbeb-055b-4c85-aebe-d4856252042a","Type":"ContainerStarted","Data":"b3847b42ee22afd9bfc6fba56819fbc1926713e8649882870ae8020ee581d8dc"} Nov 03 08:39:35.304057 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:35.303977 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jrqm5" event={"ID":"57491634-c53c-4571-854c-ece83d3f09a2","Type":"ContainerStarted","Data":"bafcd9d5a16abd87638eeb05e4ed76848566b24b68f6a0c1d3a58ca79e0843b5"} Nov 03 08:39:35.306122 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:35.306094 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-vrpjd" event={"ID":"283e4bdb-5662-4b18-8775-7c86dd4323f3","Type":"ContainerStarted","Data":"625940881e4004a06cb37ec530349840f815d5cb9ed1292170988ee3c91a9484"} Nov 03 08:39:35.353822 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:35.353794 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:35.354160 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:35.354142 5015 scope.go:122] "RemoveContainer" containerID="3a14aa4266aa2c82a4bd1d02c088d5f777afc4bc2b6d1114c870936fb1dc645b" Nov 03 08:39:35.354363 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:35.354341 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b2jvc_openshift-console-operator(a21ac76c-2765-417e-832a-32e140173e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podUID="a21ac76c-2765-417e-832a-32e140173e03" Nov 03 08:39:35.632590 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:35.632426 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:35.638027 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:35.637994 5015 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 08:39:36.143824 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.143784 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:36.144011 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.143843 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:36.144011 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.143976 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:36.144134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.144048 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:36.144134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.144085 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:36.144134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.144110 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:36.146685 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.146647 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63810bfe-e55b-4524-bde4-8f072b4b63a3-metrics-certs\") pod \"network-metrics-daemon-q7vbp\" (UID: \"63810bfe-e55b-4524-bde4-8f072b4b63a3\") " pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:36.146801 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.146726 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/587a6bc5-67cb-469b-b4b9-b35da4fb382b-metrics-tls\") pod \"dns-default-6rpvp\" (UID: \"587a6bc5-67cb-469b-b4b9-b35da4fb382b\") " pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:36.147129 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.147107 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"image-registry-6d6db59d9d-pk78q\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:36.147269 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.147108 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bebbc2e-c649-4f7f-bef3-0a8f158d6cae-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jgcwt\" (UID: \"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:36.147269 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.147175 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"image-registry-64c88f488c-xq5g2\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:36.147269 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.147256 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bfc17c-8e18-473a-a8fe-e386112cfaa4-metrics-certs\") pod \"router-default-6b64c9d49-xhzz8\" (UID: \"70bfc17c-8e18-473a-a8fe-e386112cfaa4\") " pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:36.311755 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.311713 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pc6m2" event={"ID":"19c1fbeb-055b-4c85-aebe-d4856252042a","Type":"ContainerStarted","Data":"9add2e2eecd09af2aea737f8b0b0c74a190e2b785fb9ca527a42cc77cb45a746"} Nov 03 08:39:36.396728 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.396652 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:36.397747 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.397728 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" Nov 03 08:39:36.397816 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.397797 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:36.406661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.406639 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vbp" Nov 03 08:39:36.409380 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.409363 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:36.410504 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.410462 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:36.518637 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.518600 5015 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp76w_1330da31-fcb6-4c7b-9db7-0d4d96262234/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:36.805340 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.805310 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b64c9d49-xhzz8"] Nov 03 08:39:36.830306 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:36.830261 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bfc17c_8e18_473a_a8fe_e386112cfaa4.slice/crio-4025a35f64d3d5f88c738c3c83d1e904b38a23a295efa21908cdd41bc2a07b85 WatchSource:0}: Error finding container 4025a35f64d3d5f88c738c3c83d1e904b38a23a295efa21908cdd41bc2a07b85: Status 404 returned error can't find the container with id 4025a35f64d3d5f88c738c3c83d1e904b38a23a295efa21908cdd41bc2a07b85 Nov 03 08:39:36.837973 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.837950 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt"] Nov 03 08:39:36.856159 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.856130 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64c88f488c-xq5g2"] Nov 03 08:39:36.878797 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:36.878770 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1845f9_4e7a_40b2_922a_95e1cb82eeea.slice/crio-e0d8f792c44d6c6a55b5391a333e53fcfd2841a31f3c057c6d1f433463599fca WatchSource:0}: Error finding container e0d8f792c44d6c6a55b5391a333e53fcfd2841a31f3c057c6d1f433463599fca: Status 404 returned error can't find the container with id e0d8f792c44d6c6a55b5391a333e53fcfd2841a31f3c057c6d1f433463599fca Nov 03 08:39:36.889428 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.889402 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q7vbp"] Nov 03 08:39:36.889990 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.889972 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6rpvp"] Nov 03 08:39:36.892212 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.892179 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d6db59d9d-pk78q"] Nov 03 08:39:36.894264 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:36.894234 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587a6bc5_67cb_469b_b4b9_b35da4fb382b.slice/crio-0845ad00c03df3fe62f761d36a5dc60826e380c28f34765412775ca73c031b6f WatchSource:0}: Error finding container 0845ad00c03df3fe62f761d36a5dc60826e380c28f34765412775ca73c031b6f: Status 404 returned error can't find the container with id 0845ad00c03df3fe62f761d36a5dc60826e380c28f34765412775ca73c031b6f Nov 03 08:39:36.896505 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:36.896318 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63810bfe_e55b_4524_bde4_8f072b4b63a3.slice/crio-9c2da36f8608572bf76425160455a5388089e087cda71b94c06ec5ada7aa03a9 WatchSource:0}: Error finding container 9c2da36f8608572bf76425160455a5388089e087cda71b94c06ec5ada7aa03a9: Status 404 returned error can't find the container with id 9c2da36f8608572bf76425160455a5388089e087cda71b94c06ec5ada7aa03a9 Nov 03 08:39:36.929278 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:36.929242 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e56142_8762_447d_af25_d0ba1b09b530.slice/crio-6fc0fbfb074c075d2adf17f12f5a0d894c6d8b5a67a597b8edd7defbbe651670 WatchSource:0}: Error finding container 6fc0fbfb074c075d2adf17f12f5a0d894c6d8b5a67a597b8edd7defbbe651670: Status 404 returned error can't find the container with id 6fc0fbfb074c075d2adf17f12f5a0d894c6d8b5a67a597b8edd7defbbe651670 Nov 03 08:39:36.934346 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:36.934322 5015 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hp76w_1330da31-fcb6-4c7b-9db7-0d4d96262234/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:37.316627 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.316584 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6rpvp" event={"ID":"587a6bc5-67cb-469b-b4b9-b35da4fb382b","Type":"ContainerStarted","Data":"0845ad00c03df3fe62f761d36a5dc60826e380c28f34765412775ca73c031b6f"} Nov 03 08:39:37.318258 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.318225 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b64c9d49-xhzz8" event={"ID":"70bfc17c-8e18-473a-a8fe-e386112cfaa4","Type":"ContainerStarted","Data":"5f260e72bf16888449a2e4097b7f8389cd6ecc90bf5ad9984000dc4390095d8e"} Nov 03 08:39:37.318389 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.318270 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b64c9d49-xhzz8" event={"ID":"70bfc17c-8e18-473a-a8fe-e386112cfaa4","Type":"ContainerStarted","Data":"4025a35f64d3d5f88c738c3c83d1e904b38a23a295efa21908cdd41bc2a07b85"} Nov 03 08:39:37.319413 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.319382 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7vbp" event={"ID":"63810bfe-e55b-4524-bde4-8f072b4b63a3","Type":"ContainerStarted","Data":"9c2da36f8608572bf76425160455a5388089e087cda71b94c06ec5ada7aa03a9"} Nov 03 08:39:37.320923 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.320897 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" event={"ID":"a0e56142-8762-447d-af25-d0ba1b09b530","Type":"ContainerStarted","Data":"3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e"} Nov 03 08:39:37.321048 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.320939 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" event={"ID":"a0e56142-8762-447d-af25-d0ba1b09b530","Type":"ContainerStarted","Data":"6fc0fbfb074c075d2adf17f12f5a0d894c6d8b5a67a597b8edd7defbbe651670"} Nov 03 08:39:37.322520 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.322497 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" event={"ID":"1b1845f9-4e7a-40b2-922a-95e1cb82eeea","Type":"ContainerStarted","Data":"c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259"} Nov 03 08:39:37.322612 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.322537 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" event={"ID":"1b1845f9-4e7a-40b2-922a-95e1cb82eeea","Type":"ContainerStarted","Data":"e0d8f792c44d6c6a55b5391a333e53fcfd2841a31f3c057c6d1f433463599fca"} Nov 03 08:39:37.326995 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.326975 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" event={"ID":"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae","Type":"ContainerStarted","Data":"0ac93961e1c3fb321d9a314e8eb48afc597780a19b4425e878c91acc2aee9c02"} Nov 03 08:39:37.397627 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.397591 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:37.400058 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.400036 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:37.642500 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.642399 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:39:37.696686 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:37.696662 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hp76w" Nov 03 08:39:38.148707 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:38.148671 5015 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:38.149733 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:38.149136 5015 scope.go:122] "RemoveContainer" containerID="3a14aa4266aa2c82a4bd1d02c088d5f777afc4bc2b6d1114c870936fb1dc645b" Nov 03 08:39:38.149733 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:38.149360 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b2jvc_openshift-console-operator(a21ac76c-2765-417e-832a-32e140173e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podUID="a21ac76c-2765-417e-832a-32e140173e03" Nov 03 08:39:38.335575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:38.335549 5015 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 08:39:38.714172 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:38.714088 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:38.715638 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:38.715607 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6b64c9d49-xhzz8" Nov 03 08:39:39.201360 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.201335 5015 scope.go:122] "RemoveContainer" containerID="3a14aa4266aa2c82a4bd1d02c088d5f777afc4bc2b6d1114c870936fb1dc645b" Nov 03 08:39:39.340983 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.340955 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7vbp" event={"ID":"63810bfe-e55b-4524-bde4-8f072b4b63a3","Type":"ContainerStarted","Data":"04f7936dab18309575001c6607af6dca6d892a10ec85564d28891558e894d9d1"} Nov 03 08:39:39.341345 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.341018 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7vbp" event={"ID":"63810bfe-e55b-4524-bde4-8f072b4b63a3","Type":"ContainerStarted","Data":"0dc2149fa41236fda8e2ad3654ecf1394634dfd1525ec072f9ee326deca77ffa"} Nov 03 08:39:39.342451 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.342433 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 08:39:39.342889 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.342871 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/2.log" Nov 03 08:39:39.342950 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.342908 5015 generic.go:356] "Generic (PLEG): container finished" podID="a21ac76c-2765-417e-832a-32e140173e03" containerID="c4978aee24432efb3ad85fb8c9916f045d0567b70f723232a230031de56c1c4d" exitCode=255 Nov 03 08:39:39.343010 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.342988 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" event={"ID":"a21ac76c-2765-417e-832a-32e140173e03","Type":"ContainerDied","Data":"c4978aee24432efb3ad85fb8c9916f045d0567b70f723232a230031de56c1c4d"} Nov 03 08:39:39.343059 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.343051 5015 scope.go:122] "RemoveContainer" containerID="3a14aa4266aa2c82a4bd1d02c088d5f777afc4bc2b6d1114c870936fb1dc645b" Nov 03 08:39:39.343373 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.343353 5015 scope.go:122] "RemoveContainer" containerID="c4978aee24432efb3ad85fb8c9916f045d0567b70f723232a230031de56c1c4d" Nov 03 08:39:39.343611 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:39.343586 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b2jvc_openshift-console-operator(a21ac76c-2765-417e-832a-32e140173e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podUID="a21ac76c-2765-417e-832a-32e140173e03" Nov 03 08:39:39.345063 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.345039 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6rpvp" event={"ID":"587a6bc5-67cb-469b-b4b9-b35da4fb382b","Type":"ContainerStarted","Data":"73c90981b9a6d8c25521116ac398fa0d76737653aa5b79fba78a22df4f101d89"} Nov 03 08:39:39.345161 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.345084 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6rpvp" event={"ID":"587a6bc5-67cb-469b-b4b9-b35da4fb382b","Type":"ContainerStarted","Data":"201640d3ffbe5bcc61db1f32a8338c8fb3bdadb496a0c9a3bd8b77c793edb48e"} Nov 03 08:39:39.345418 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:39.345402 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:40.109770 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:40.109733 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:40.295976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:40.295948 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fnh7l" Nov 03 08:39:40.351794 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:40.351762 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 08:39:40.353638 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:40.353610 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" event={"ID":"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae","Type":"ContainerStarted","Data":"6f30caa859a05aaf3ef4818fcf622f9b06e2ff6215827da75479931c3f0c443d"} Nov 03 08:39:40.353757 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:40.353666 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jgcwt" event={"ID":"7bebbc2e-c649-4f7f-bef3-0a8f158d6cae","Type":"ContainerStarted","Data":"5b8cbba5ada26d620cd4ea6fb931dba1ffd9a85e80e7eb8f953f489c570ace7b"} Nov 03 08:39:45.354027 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:45.353991 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:45.354521 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:45.354334 5015 scope.go:122] "RemoveContainer" containerID="c4978aee24432efb3ad85fb8c9916f045d0567b70f723232a230031de56c1c4d" Nov 03 08:39:45.354567 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:45.354532 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b2jvc_openshift-console-operator(a21ac76c-2765-417e-832a-32e140173e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podUID="a21ac76c-2765-417e-832a-32e140173e03" Nov 03 08:39:46.899811 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:46.899770 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:39:47.616347 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:47.616308 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64c88f488c-xq5g2"] Nov 03 08:39:48.149457 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:48.149415 5015 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:39:48.149891 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:48.149877 5015 scope.go:122] "RemoveContainer" containerID="c4978aee24432efb3ad85fb8c9916f045d0567b70f723232a230031de56c1c4d" Nov 03 08:39:48.150084 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:39:48.150064 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-b2jvc_openshift-console-operator(a21ac76c-2765-417e-832a-32e140173e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podUID="a21ac76c-2765-417e-832a-32e140173e03" Nov 03 08:39:49.948457 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:49.948423 5015 patch_prober.go:29] interesting pod/image-registry-6d6db59d9d-pk78q container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 08:39:49.948973 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:49.948491 5015 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" podUID="a0e56142-8762-447d-af25-d0ba1b09b530" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:39:50.359314 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:50.359281 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6rpvp" Nov 03 08:39:57.516672 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.516637 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-frljt"] Nov 03 08:39:57.567989 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.567956 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.621826 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.621797 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:39:57.676788 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.676756 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 08:39:57.677069 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.677052 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 08:39:57.677148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.677085 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 08:39:57.677209 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.677180 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h97f4\"" Nov 03 08:39:57.677262 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.677214 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 08:39:57.726984 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.726958 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-wtmp\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.727125 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.726992 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-tls\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.727125 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.727020 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dafdc498-1da7-4acb-9a07-a95413ed93d0-root\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.727125 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.727054 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dafdc498-1da7-4acb-9a07-a95413ed93d0-metrics-client-ca\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.727273 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.727142 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.727273 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.727183 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dafdc498-1da7-4acb-9a07-a95413ed93d0-sys\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.727375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.727306 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptb26\" (UniqueName: \"kubernetes.io/projected/dafdc498-1da7-4acb-9a07-a95413ed93d0-kube-api-access-ptb26\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.727375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.727349 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-accelerators-collector-config\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.727497 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.727422 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-textfile\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828172 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828092 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dafdc498-1da7-4acb-9a07-a95413ed93d0-root\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828172 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828124 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dafdc498-1da7-4acb-9a07-a95413ed93d0-metrics-client-ca\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828172 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828154 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828177 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dafdc498-1da7-4acb-9a07-a95413ed93d0-sys\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828216 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dafdc498-1da7-4acb-9a07-a95413ed93d0-root\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828244 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dafdc498-1da7-4acb-9a07-a95413ed93d0-sys\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828267 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptb26\" (UniqueName: \"kubernetes.io/projected/dafdc498-1da7-4acb-9a07-a95413ed93d0-kube-api-access-ptb26\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828297 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-accelerators-collector-config\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828327 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-textfile\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828373 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-wtmp\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828404 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-tls\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828806 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828715 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-wtmp\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828852 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828838 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dafdc498-1da7-4acb-9a07-a95413ed93d0-metrics-client-ca\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828914 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828895 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-textfile\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.828962 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.828944 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-accelerators-collector-config\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.831320 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.831298 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-tls\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.831420 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.831306 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dafdc498-1da7-4acb-9a07-a95413ed93d0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.888633 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.888605 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptb26\" (UniqueName: \"kubernetes.io/projected/dafdc498-1da7-4acb-9a07-a95413ed93d0-kube-api-access-ptb26\") pod \"node-exporter-frljt\" (UID: \"dafdc498-1da7-4acb-9a07-a95413ed93d0\") " pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.921572 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:57.921542 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-frljt" Nov 03 08:39:57.956644 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:39:57.956614 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafdc498_1da7_4acb_9a07_a95413ed93d0.slice/crio-52a649f5c8ae335241b99a8aadd888c58cd3ccac38596f5b93aa988cd30e2f62 WatchSource:0}: Error finding container 52a649f5c8ae335241b99a8aadd888c58cd3ccac38596f5b93aa988cd30e2f62: Status 404 returned error can't find the container with id 52a649f5c8ae335241b99a8aadd888c58cd3ccac38596f5b93aa988cd30e2f62 Nov 03 08:39:58.423027 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:58.422993 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frljt" event={"ID":"dafdc498-1da7-4acb-9a07-a95413ed93d0","Type":"ContainerStarted","Data":"52a649f5c8ae335241b99a8aadd888c58cd3ccac38596f5b93aa988cd30e2f62"} Nov 03 08:39:59.351056 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:39:59.351015 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:40:00.438696 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:00.438658 5015 generic.go:356] "Generic (PLEG): container finished" podID="dafdc498-1da7-4acb-9a07-a95413ed93d0" containerID="97d108bfb5f53bd9fc481c7d82bfc5875a083cbc7be3b82107f4b36935e60030" exitCode=0 Nov 03 08:40:00.439151 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:00.438747 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frljt" event={"ID":"dafdc498-1da7-4acb-9a07-a95413ed93d0","Type":"ContainerDied","Data":"97d108bfb5f53bd9fc481c7d82bfc5875a083cbc7be3b82107f4b36935e60030"} Nov 03 08:40:01.017226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:01.017194 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4cbq5" Nov 03 08:40:01.446024 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:01.445933 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frljt" event={"ID":"dafdc498-1da7-4acb-9a07-a95413ed93d0","Type":"ContainerStarted","Data":"962bb1ed5cecd93c69c5e26ad4b6ee42e44ac060ad5a9cf73ae756f2f143fbe3"} Nov 03 08:40:01.446024 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:01.445992 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frljt" event={"ID":"dafdc498-1da7-4acb-9a07-a95413ed93d0","Type":"ContainerStarted","Data":"6d7a40640a250a407b9a21e15bf61afcda59706bd4be384560973b0a30a5116c"} Nov 03 08:40:01.565166 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:01.565117 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-frljt" podStartSLOduration=2.537080447 podStartE2EDuration="4.565102636s" totalImagesPullingTime="1.939916641s" totalInitContainerRuntime="88.105548ms" isStatefulPod=true podCreationTimestamp="2026-11-03 08:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:01.564617899 +0000 UTC m=+34.489560128" watchObservedRunningTime="2026-11-03 08:40:01.565102636 +0000 UTC m=+34.490044849" Nov 03 08:40:04.196615 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:04.196575 5015 scope.go:122] "RemoveContainer" containerID="c4978aee24432efb3ad85fb8c9916f045d0567b70f723232a230031de56c1c4d" Nov 03 08:40:04.459823 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:04.459739 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 08:40:04.459998 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:04.459830 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" event={"ID":"a21ac76c-2765-417e-832a-32e140173e03","Type":"ContainerStarted","Data":"ce6c5116ab0de19ba5bd0116fe97e74153e105975fe2f791e8243fd2e1aa3260"} Nov 03 08:40:04.460173 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:04.460154 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:40:04.461105 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:04.461085 5015 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-b2jvc container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Nov 03 08:40:04.461169 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:04.461124 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" podUID="a21ac76c-2765-417e-832a-32e140173e03" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Nov 03 08:40:05.467942 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:05.467858 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-b2jvc" Nov 03 08:40:10.720490 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:10.720453 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d6db59d9d-pk78q"] Nov 03 08:40:12.637336 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.637282 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" podUID="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" containerName="registry" containerID="cri-o://c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259" gracePeriod=30 Nov 03 08:40:12.885150 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.885127 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:40:12.958899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.958827 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") pod \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " Nov 03 08:40:12.958899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.958867 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-ca-trust-extracted\") pod \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " Nov 03 08:40:12.959109 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.958906 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-certificates\") pod \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " Nov 03 08:40:12.959109 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.959034 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-bound-sa-token\") pod \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " Nov 03 08:40:12.959109 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.959089 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-image-registry-private-configuration\") pod \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " Nov 03 08:40:12.959255 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.959151 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-kube-api-access-z27j2\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-kube-api-access-z27j2\") pod \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " Nov 03 08:40:12.959255 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.959227 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-installation-pull-secrets\") pod \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " Nov 03 08:40:12.959345 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.959272 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-trusted-ca\") pod \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\" (UID: \"1b1845f9-4e7a-40b2-922a-95e1cb82eeea\") " Nov 03 08:40:12.959345 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.959276 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-certificates" pod "1b1845f9-4e7a-40b2-922a-95e1cb82eeea" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:12.961880 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.961080 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-trusted-ca" pod "1b1845f9-4e7a-40b2-922a-95e1cb82eeea" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:12.964188 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.962662 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls" pod "1b1845f9-4e7a-40b2-922a-95e1cb82eeea" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:40:12.964188 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.964164 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-installation-pull-secrets" pod "1b1845f9-4e7a-40b2-922a-95e1cb82eeea" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:12.964395 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.964281 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-bound-sa-token" pod "1b1845f9-4e7a-40b2-922a-95e1cb82eeea" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:40:12.964395 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.964319 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-image-registry-private-configuration" pod "1b1845f9-4e7a-40b2-922a-95e1cb82eeea" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:12.965969 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.965948 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-kube-api-access-z27j2" pod "1b1845f9-4e7a-40b2-922a-95e1cb82eeea" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea"). InnerVolumeSpecName "kube-api-access-z27j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:40:12.970292 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:12.970265 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-ca-trust-extracted" pod "1b1845f9-4e7a-40b2-922a-95e1cb82eeea" (UID: "1b1845f9-4e7a-40b2-922a-95e1cb82eeea"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:40:13.061245 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.061216 5015 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-trusted-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:13.061245 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.061242 5015 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:13.061245 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.061252 5015 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-ca-trust-extracted\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:13.061461 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.061265 5015 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-registry-certificates\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:13.061461 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.061282 5015 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-bound-sa-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:13.061461 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.061297 5015 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-image-registry-private-configuration\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:13.061461 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.061312 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z27j2\" (UniqueName: \"kubernetes.io/projected/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-kube-api-access-z27j2\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:13.061461 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.061326 5015 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b1845f9-4e7a-40b2-922a-95e1cb82eeea-installation-pull-secrets\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:13.494763 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.494722 5015 generic.go:356] "Generic (PLEG): container finished" podID="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" containerID="c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259" exitCode=0 Nov 03 08:40:13.494763 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.494747 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" Nov 03 08:40:13.494966 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.494770 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" event={"ID":"1b1845f9-4e7a-40b2-922a-95e1cb82eeea","Type":"ContainerDied","Data":"c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259"} Nov 03 08:40:13.494966 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.494818 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c88f488c-xq5g2" event={"ID":"1b1845f9-4e7a-40b2-922a-95e1cb82eeea","Type":"ContainerDied","Data":"e0d8f792c44d6c6a55b5391a333e53fcfd2841a31f3c057c6d1f433463599fca"} Nov 03 08:40:13.494966 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.494838 5015 scope.go:122] "RemoveContainer" containerID="c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259" Nov 03 08:40:13.503827 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.503805 5015 scope.go:122] "RemoveContainer" containerID="c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259" Nov 03 08:40:13.504136 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:40:13.504110 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259\": container with ID starting with c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259 not found: ID does not exist" containerID="c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259" Nov 03 08:40:13.504231 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.504148 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259"} err="failed to get container status \"c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259\": rpc error: code = NotFound desc = could not find container \"c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259\": container with ID starting with c6af70254cbd0e8bd879fc45abbd072eafdee4bce50bb61611a329995c653259 not found: ID does not exist" Nov 03 08:40:13.665272 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.665241 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64c88f488c-xq5g2"] Nov 03 08:40:13.715771 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:13.715741 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64c88f488c-xq5g2"] Nov 03 08:40:15.201799 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:15.201768 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" path="/var/lib/kubelet/pods/1b1845f9-4e7a-40b2-922a-95e1cb82eeea/volumes" Nov 03 08:40:16.903914 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:40:16.903847 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:40:35.740567 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:35.740530 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" podUID="a0e56142-8762-447d-af25-d0ba1b09b530" containerName="registry" containerID="cri-o://3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e" gracePeriod=30 Nov 03 08:40:35.995906 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:35.995852 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:40:36.046908 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.046882 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") pod \"a0e56142-8762-447d-af25-d0ba1b09b530\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " Nov 03 08:40:36.047044 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.046927 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-kube-api-access-p74hn\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-kube-api-access-p74hn\") pod \"a0e56142-8762-447d-af25-d0ba1b09b530\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " Nov 03 08:40:36.047044 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.046954 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e56142-8762-447d-af25-d0ba1b09b530-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0e56142-8762-447d-af25-d0ba1b09b530-ca-trust-extracted\") pod \"a0e56142-8762-447d-af25-d0ba1b09b530\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " Nov 03 08:40:36.047044 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.046976 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-registry-certificates\") pod \"a0e56142-8762-447d-af25-d0ba1b09b530\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " Nov 03 08:40:36.047044 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.047013 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-bound-sa-token\") pod \"a0e56142-8762-447d-af25-d0ba1b09b530\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " Nov 03 08:40:36.047254 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.047067 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-image-registry-private-configuration\") pod \"a0e56142-8762-447d-af25-d0ba1b09b530\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " Nov 03 08:40:36.047254 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.047103 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-trusted-ca\") pod \"a0e56142-8762-447d-af25-d0ba1b09b530\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " Nov 03 08:40:36.047254 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.047141 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-installation-pull-secrets\") pod \"a0e56142-8762-447d-af25-d0ba1b09b530\" (UID: \"a0e56142-8762-447d-af25-d0ba1b09b530\") " Nov 03 08:40:36.047510 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.047454 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-registry-certificates" pod "a0e56142-8762-447d-af25-d0ba1b09b530" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:36.048052 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.048018 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-trusted-ca" pod "a0e56142-8762-447d-af25-d0ba1b09b530" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:36.049367 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.049320 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls" pod "a0e56142-8762-447d-af25-d0ba1b09b530" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:40:36.049674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.049650 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-kube-api-access-p74hn" pod "a0e56142-8762-447d-af25-d0ba1b09b530" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530"). InnerVolumeSpecName "kube-api-access-p74hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:40:36.049903 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.049875 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-image-registry-private-configuration" pod "a0e56142-8762-447d-af25-d0ba1b09b530" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:36.050008 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.049981 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-bound-sa-token" pod "a0e56142-8762-447d-af25-d0ba1b09b530" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:40:36.050008 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.049994 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-installation-pull-secrets" pod "a0e56142-8762-447d-af25-d0ba1b09b530" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:36.054141 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.054121 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e56142-8762-447d-af25-d0ba1b09b530-ca-trust-extracted" pod "a0e56142-8762-447d-af25-d0ba1b09b530" (UID: "a0e56142-8762-447d-af25-d0ba1b09b530"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:40:36.148410 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.148379 5015 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-registry-certificates\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:36.148410 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.148407 5015 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-bound-sa-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:36.148410 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.148417 5015 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-image-registry-private-configuration\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:36.148655 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.148427 5015 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0e56142-8762-447d-af25-d0ba1b09b530-trusted-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:36.148655 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.148437 5015 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a0e56142-8762-447d-af25-d0ba1b09b530-installation-pull-secrets\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:36.148655 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.148445 5015 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-registry-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:36.148655 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.148453 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74hn\" (UniqueName: \"kubernetes.io/projected/a0e56142-8762-447d-af25-d0ba1b09b530-kube-api-access-p74hn\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:36.148655 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.148461 5015 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a0e56142-8762-447d-af25-d0ba1b09b530-ca-trust-extracted\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:36.586890 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.586855 5015 generic.go:356] "Generic (PLEG): container finished" podID="a0e56142-8762-447d-af25-d0ba1b09b530" containerID="3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e" exitCode=0 Nov 03 08:40:36.587057 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.586906 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" Nov 03 08:40:36.587057 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.586924 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" event={"ID":"a0e56142-8762-447d-af25-d0ba1b09b530","Type":"ContainerDied","Data":"3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e"} Nov 03 08:40:36.587057 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.586973 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6db59d9d-pk78q" event={"ID":"a0e56142-8762-447d-af25-d0ba1b09b530","Type":"ContainerDied","Data":"6fc0fbfb074c075d2adf17f12f5a0d894c6d8b5a67a597b8edd7defbbe651670"} Nov 03 08:40:36.587057 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.587000 5015 scope.go:122] "RemoveContainer" containerID="3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e" Nov 03 08:40:36.595976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.595958 5015 scope.go:122] "RemoveContainer" containerID="3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e" Nov 03 08:40:36.596256 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:40:36.596234 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e\": container with ID starting with 3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e not found: ID does not exist" containerID="3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e" Nov 03 08:40:36.596310 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.596267 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e"} err="failed to get container status \"3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e\": rpc error: code = NotFound desc = could not find container \"3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e\": container with ID starting with 3b954f6f78aeced20408a83c320dc1a4c770d1534831e5e5a81a356ba123f79e not found: ID does not exist" Nov 03 08:40:36.757148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.755534 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d6db59d9d-pk78q"] Nov 03 08:40:36.805344 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:36.805314 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d6db59d9d-pk78q"] Nov 03 08:40:37.201925 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:37.201890 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e56142-8762-447d-af25-d0ba1b09b530" path="/var/lib/kubelet/pods/a0e56142-8762-447d-af25-d0ba1b09b530/volumes" Nov 03 08:40:43.617575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:43.617540 5015 generic.go:356] "Generic (PLEG): container finished" podID="16cb59f2-9d9f-4904-9fc8-bed9111deecd" containerID="b5bdf845da4aca9cac4741b383cbb241817975e2d32455659b41e5d5ee1614b8" exitCode=0 Nov 03 08:40:43.617956 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:43.617608 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" event={"ID":"16cb59f2-9d9f-4904-9fc8-bed9111deecd","Type":"ContainerDied","Data":"b5bdf845da4aca9cac4741b383cbb241817975e2d32455659b41e5d5ee1614b8"} Nov 03 08:40:43.617956 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:43.617936 5015 scope.go:122] "RemoveContainer" containerID="b5bdf845da4aca9cac4741b383cbb241817975e2d32455659b41e5d5ee1614b8" Nov 03 08:40:44.623001 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:44.622968 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s5kxx" event={"ID":"16cb59f2-9d9f-4904-9fc8-bed9111deecd","Type":"ContainerStarted","Data":"35a999ace8c8e56cb07a2560179cda343a87239585e7767a05447d06e8975f12"} Nov 03 08:40:46.900070 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:40:46.900027 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:40:48.640524 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:48.640489 5015 generic.go:356] "Generic (PLEG): container finished" podID="df61dbd1-26b9-4d1b-847c-785687d935ab" containerID="777a6be2956e0afa3006eb9183f62eb0a34872a9553d5dce2c1808daa768fcb1" exitCode=0 Nov 03 08:40:48.640900 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:48.640529 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" event={"ID":"df61dbd1-26b9-4d1b-847c-785687d935ab","Type":"ContainerDied","Data":"777a6be2956e0afa3006eb9183f62eb0a34872a9553d5dce2c1808daa768fcb1"} Nov 03 08:40:48.640900 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:48.640838 5015 scope.go:122] "RemoveContainer" containerID="777a6be2956e0afa3006eb9183f62eb0a34872a9553d5dce2c1808daa768fcb1" Nov 03 08:40:49.646676 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:49.646640 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ck58r" event={"ID":"df61dbd1-26b9-4d1b-847c-785687d935ab","Type":"ContainerStarted","Data":"a2b74c3b15fb7587e11256e5fc046b2c955ad032b61351509359238ded53d10b"} Nov 03 08:40:53.663230 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:53.663196 5015 generic.go:356] "Generic (PLEG): container finished" podID="22edddd5-3b05-4942-910a-651d537867c8" containerID="a401dae88d6eb1871e59e25194d85f40ab0ce4e0864f2441649738ebf38b5804" exitCode=0 Nov 03 08:40:53.663641 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:53.663270 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-b4ggb" event={"ID":"22edddd5-3b05-4942-910a-651d537867c8","Type":"ContainerDied","Data":"a401dae88d6eb1871e59e25194d85f40ab0ce4e0864f2441649738ebf38b5804"} Nov 03 08:40:53.663641 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:53.663638 5015 scope.go:122] "RemoveContainer" containerID="a401dae88d6eb1871e59e25194d85f40ab0ce4e0864f2441649738ebf38b5804" Nov 03 08:40:54.628887 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:54.628848 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-llp7m-mp4tf"] Nov 03 08:40:54.629070 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:54.629036 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-llp7m-mp4tf" podUID="e0106338-5b18-4bbf-abfe-f2eee3213c8c" containerName="insights-gathering" containerID="cri-o://354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f" gracePeriod=30 Nov 03 08:40:54.669634 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:54.669594 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-b4ggb" event={"ID":"22edddd5-3b05-4942-910a-651d537867c8","Type":"ContainerStarted","Data":"40a888cb67f1b5f910695dea33bd43738a7c9bd71558e0436b82cb4d2e0b0ccc"} Nov 03 08:40:54.877775 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:54.877750 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:40:55.005991 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.005965 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0106338-5b18-4bbf-abfe-f2eee3213c8c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0106338-5b18-4bbf-abfe-f2eee3213c8c-archives-path\") pod \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " Nov 03 08:40:55.006139 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.006034 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-service-ca-bundle\") pod \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " Nov 03 08:40:55.006139 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.006072 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0106338-5b18-4bbf-abfe-f2eee3213c8c-kube-api-access-wdt5w\" (UniqueName: \"kubernetes.io/projected/e0106338-5b18-4bbf-abfe-f2eee3213c8c-kube-api-access-wdt5w\") pod \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " Nov 03 08:40:55.006211 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.006137 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-trusted-ca-bundle\") pod \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\" (UID: \"e0106338-5b18-4bbf-abfe-f2eee3213c8c\") " Nov 03 08:40:55.006349 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.006324 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0106338-5b18-4bbf-abfe-f2eee3213c8c-archives-path" pod "e0106338-5b18-4bbf-abfe-f2eee3213c8c" (UID: "e0106338-5b18-4bbf-abfe-f2eee3213c8c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:40:55.006542 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.006520 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-service-ca-bundle" pod "e0106338-5b18-4bbf-abfe-f2eee3213c8c" (UID: "e0106338-5b18-4bbf-abfe-f2eee3213c8c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:55.006613 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.006567 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-trusted-ca-bundle" pod "e0106338-5b18-4bbf-abfe-f2eee3213c8c" (UID: "e0106338-5b18-4bbf-abfe-f2eee3213c8c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:55.008542 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.008515 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0106338-5b18-4bbf-abfe-f2eee3213c8c-kube-api-access-wdt5w" pod "e0106338-5b18-4bbf-abfe-f2eee3213c8c" (UID: "e0106338-5b18-4bbf-abfe-f2eee3213c8c"). InnerVolumeSpecName "kube-api-access-wdt5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:40:55.071338 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.071307 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-r6xbr-hw6qv"] Nov 03 08:40:55.071624 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.071610 5015 cpu_manager.go:414] "containerMap: removing container" podUID="e0106338-5b18-4bbf-abfe-f2eee3213c8c" containerName="insights-gathering" Nov 03 08:40:55.071674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.071626 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0106338-5b18-4bbf-abfe-f2eee3213c8c" containerName="insights-gathering" Nov 03 08:40:55.071674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.071643 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a0e56142-8762-447d-af25-d0ba1b09b530" containerName="registry" Nov 03 08:40:55.071674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.071648 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e56142-8762-447d-af25-d0ba1b09b530" containerName="registry" Nov 03 08:40:55.071674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.071665 5015 cpu_manager.go:414] "containerMap: removing container" podUID="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" containerName="registry" Nov 03 08:40:55.071674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.071669 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" containerName="registry" Nov 03 08:40:55.071824 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.071727 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e56142-8762-447d-af25-d0ba1b09b530" containerName="registry" Nov 03 08:40:55.071824 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.071740 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1845f9-4e7a-40b2-922a-95e1cb82eeea" containerName="registry" Nov 03 08:40:55.071824 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.071752 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0106338-5b18-4bbf-abfe-f2eee3213c8c" containerName="insights-gathering" Nov 03 08:40:55.074587 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.074571 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.107250 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.107227 5015 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-trusted-ca-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:55.107250 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.107250 5015 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e0106338-5b18-4bbf-abfe-f2eee3213c8c-archives-path\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:55.107387 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.107259 5015 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0106338-5b18-4bbf-abfe-f2eee3213c8c-service-ca-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:55.107387 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.107270 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdt5w\" (UniqueName: \"kubernetes.io/projected/e0106338-5b18-4bbf-abfe-f2eee3213c8c-kube-api-access-wdt5w\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:55.133177 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.133149 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-r6xbr-hw6qv"] Nov 03 08:40:55.208691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.208630 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d5cbd134-00f7-43d1-9890-960de193c02f-archives-path\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.208839 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.208743 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-trusted-ca-bundle\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.208839 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.208770 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqq99\" (UniqueName: \"kubernetes.io/projected/d5cbd134-00f7-43d1-9890-960de193c02f-kube-api-access-cqq99\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.208839 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.208800 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-service-ca-bundle\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.309415 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.309337 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-trusted-ca-bundle\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.309415 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.309370 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqq99\" (UniqueName: \"kubernetes.io/projected/d5cbd134-00f7-43d1-9890-960de193c02f-kube-api-access-cqq99\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.309415 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.309401 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-service-ca-bundle\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.309666 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.309496 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d5cbd134-00f7-43d1-9890-960de193c02f-archives-path\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.309952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.309930 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d5cbd134-00f7-43d1-9890-960de193c02f-archives-path\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.310165 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.310142 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-service-ca-bundle\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.310222 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.310205 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-trusted-ca-bundle\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.368968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.368940 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqq99\" (UniqueName: \"kubernetes.io/projected/d5cbd134-00f7-43d1-9890-960de193c02f-kube-api-access-cqq99\") pod \"periodic-gathering-r6xbr-hw6qv\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.385791 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.385762 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:40:55.675356 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.675248 5015 generic.go:356] "Generic (PLEG): container finished" podID="e0106338-5b18-4bbf-abfe-f2eee3213c8c" containerID="354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f" exitCode=2 Nov 03 08:40:55.675356 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.675300 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-llp7m-mp4tf" Nov 03 08:40:55.675356 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.675322 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-llp7m-mp4tf" event={"ID":"e0106338-5b18-4bbf-abfe-f2eee3213c8c","Type":"ContainerDied","Data":"354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f"} Nov 03 08:40:55.675836 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.675372 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-llp7m-mp4tf" event={"ID":"e0106338-5b18-4bbf-abfe-f2eee3213c8c","Type":"ContainerDied","Data":"5cf9fe8900d2c446a0e01afa796d2f387c7bf1ef01ea9f42c14a1843edf38c8f"} Nov 03 08:40:55.675836 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.675392 5015 scope.go:122] "RemoveContainer" containerID="354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f" Nov 03 08:40:55.683371 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.683353 5015 scope.go:122] "RemoveContainer" containerID="354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f" Nov 03 08:40:55.683701 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:40:55.683677 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f\": container with ID starting with 354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f not found: ID does not exist" containerID="354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f" Nov 03 08:40:55.683765 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.683706 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f"} err="failed to get container status \"354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f\": rpc error: code = NotFound desc = could not find container \"354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f\": container with ID starting with 354aa055212a7e333b02005d86347a70ab820f57cae8a24db74f6e3d197a009f not found: ID does not exist" Nov 03 08:40:55.758969 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.758942 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-r6xbr-hw6qv"] Nov 03 08:40:55.761761 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:40:55.761734 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cbd134_00f7_43d1_9890_960de193c02f.slice/crio-2ad370feaeb7411c3fa922bdc9515dc5ac4be48fc2c6f7f9d19e92b228795fcc WatchSource:0}: Error finding container 2ad370feaeb7411c3fa922bdc9515dc5ac4be48fc2c6f7f9d19e92b228795fcc: Status 404 returned error can't find the container with id 2ad370feaeb7411c3fa922bdc9515dc5ac4be48fc2c6f7f9d19e92b228795fcc Nov 03 08:40:55.852191 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.852162 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-llp7m-mp4tf"] Nov 03 08:40:55.863356 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:55.863326 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-llp7m-mp4tf"] Nov 03 08:40:56.683752 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:56.683718 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" event={"ID":"d5cbd134-00f7-43d1-9890-960de193c02f","Type":"ContainerStarted","Data":"d522c741304a9b2d935a7b0268642e24ec1201856fbf14ee374bd0fd5770c022"} Nov 03 08:40:56.684129 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:56.683763 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" event={"ID":"d5cbd134-00f7-43d1-9890-960de193c02f","Type":"ContainerStarted","Data":"2ad370feaeb7411c3fa922bdc9515dc5ac4be48fc2c6f7f9d19e92b228795fcc"} Nov 03 08:40:56.800937 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:56.800885 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" podStartSLOduration=1.800871707 podStartE2EDuration="1.800871707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:56.800292627 +0000 UTC m=+89.725234848" watchObservedRunningTime="2026-11-03 08:40:56.800871707 +0000 UTC m=+89.725813924" Nov 03 08:40:57.201729 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:40:57.201695 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0106338-5b18-4bbf-abfe-f2eee3213c8c" path="/var/lib/kubelet/pods/e0106338-5b18-4bbf-abfe-f2eee3213c8c/volumes" Nov 03 08:41:16.900632 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:41:16.900590 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:41:46.901249 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:41:46.901168 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:42:16.901079 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:42:16.901031 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:42:21.020092 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:21.020050 5015 generic.go:356] "Generic (PLEG): container finished" podID="d5cbd134-00f7-43d1-9890-960de193c02f" containerID="d522c741304a9b2d935a7b0268642e24ec1201856fbf14ee374bd0fd5770c022" exitCode=0 Nov 03 08:42:21.020603 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:21.020123 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" event={"ID":"d5cbd134-00f7-43d1-9890-960de193c02f","Type":"ContainerDied","Data":"d522c741304a9b2d935a7b0268642e24ec1201856fbf14ee374bd0fd5770c022"} Nov 03 08:42:22.154038 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.154015 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:42:22.304515 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.304412 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cbd134-00f7-43d1-9890-960de193c02f-kube-api-access-cqq99\" (UniqueName: \"kubernetes.io/projected/d5cbd134-00f7-43d1-9890-960de193c02f-kube-api-access-cqq99\") pod \"d5cbd134-00f7-43d1-9890-960de193c02f\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " Nov 03 08:42:22.304515 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.304496 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-service-ca-bundle\") pod \"d5cbd134-00f7-43d1-9890-960de193c02f\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " Nov 03 08:42:22.304756 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.304597 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cbd134-00f7-43d1-9890-960de193c02f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d5cbd134-00f7-43d1-9890-960de193c02f-archives-path\") pod \"d5cbd134-00f7-43d1-9890-960de193c02f\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " Nov 03 08:42:22.304756 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.304640 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-trusted-ca-bundle\") pod \"d5cbd134-00f7-43d1-9890-960de193c02f\" (UID: \"d5cbd134-00f7-43d1-9890-960de193c02f\") " Nov 03 08:42:22.304853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.304824 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-service-ca-bundle" pod "d5cbd134-00f7-43d1-9890-960de193c02f" (UID: "d5cbd134-00f7-43d1-9890-960de193c02f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:42:22.304942 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.304919 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cbd134-00f7-43d1-9890-960de193c02f-archives-path" pod "d5cbd134-00f7-43d1-9890-960de193c02f" (UID: "d5cbd134-00f7-43d1-9890-960de193c02f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:42:22.305038 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.305011 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-trusted-ca-bundle" pod "d5cbd134-00f7-43d1-9890-960de193c02f" (UID: "d5cbd134-00f7-43d1-9890-960de193c02f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:42:22.305038 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.305023 5015 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-service-ca-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:42:22.305202 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.305054 5015 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d5cbd134-00f7-43d1-9890-960de193c02f-archives-path\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:42:22.306967 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.306935 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cbd134-00f7-43d1-9890-960de193c02f-kube-api-access-cqq99" pod "d5cbd134-00f7-43d1-9890-960de193c02f" (UID: "d5cbd134-00f7-43d1-9890-960de193c02f"). InnerVolumeSpecName "kube-api-access-cqq99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:42:22.406252 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.406212 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqq99\" (UniqueName: \"kubernetes.io/projected/d5cbd134-00f7-43d1-9890-960de193c02f-kube-api-access-cqq99\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:42:22.406252 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:22.406243 5015 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5cbd134-00f7-43d1-9890-960de193c02f-trusted-ca-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:42:23.028912 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:23.028878 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" Nov 03 08:42:23.029084 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:23.028914 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r6xbr-hw6qv" event={"ID":"d5cbd134-00f7-43d1-9890-960de193c02f","Type":"ContainerDied","Data":"2ad370feaeb7411c3fa922bdc9515dc5ac4be48fc2c6f7f9d19e92b228795fcc"} Nov 03 08:42:23.029084 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:42:23.028960 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad370feaeb7411c3fa922bdc9515dc5ac4be48fc2c6f7f9d19e92b228795fcc" Nov 03 08:42:46.900982 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:42:46.900937 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:43:16.901407 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:43:16.901325 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:43:46.900657 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:43:46.900619 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:44:16.901456 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:44:16.901418 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:44:24.135828 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.135796 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf"] Nov 03 08:44:24.136289 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.136102 5015 cpu_manager.go:414] "containerMap: removing container" podUID="d5cbd134-00f7-43d1-9890-960de193c02f" containerName="insights-gathering" Nov 03 08:44:24.136289 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.136113 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cbd134-00f7-43d1-9890-960de193c02f" containerName="insights-gathering" Nov 03 08:44:24.136289 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.136179 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cbd134-00f7-43d1-9890-960de193c02f" containerName="insights-gathering" Nov 03 08:44:24.139327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.139309 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:44:24.196660 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.196625 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 08:44:24.196847 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.196666 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 08:44:24.197060 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.197040 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf"] Nov 03 08:44:24.225017 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.224986 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnnrl\" (UniqueName: \"kubernetes.io/projected/efc99055-280d-4def-833b-b73c4e1a8b8a-kube-api-access-bnnrl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:44:24.225172 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.225030 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:44:24.225213 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.225172 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:44:24.232638 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.232610 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns"] Nov 03 08:44:24.236101 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.236086 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:44:24.242038 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.242017 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-nsdl8\"" Nov 03 08:44:24.291352 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.291315 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-xm2l5\"" Nov 03 08:44:24.295151 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.295124 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns"] Nov 03 08:44:24.325650 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.325610 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnnrl\" (UniqueName: \"kubernetes.io/projected/efc99055-280d-4def-833b-b73c4e1a8b8a-kube-api-access-bnnrl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:44:24.325847 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.325667 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:44:24.325847 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.325704 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2cx\" (UniqueName: \"kubernetes.io/projected/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-kube-api-access-rl2cx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:44:24.325847 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.325734 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:44:24.325847 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.325769 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:44:24.326040 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.326017 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:44:24.326096 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.326079 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:44:24.326257 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.326241 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:44:24.332218 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.332189 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh"] Nov 03 08:44:24.335804 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.335789 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:44:24.387981 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.387907 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnnrl\" (UniqueName: \"kubernetes.io/projected/efc99055-280d-4def-833b-b73c4e1a8b8a-kube-api-access-bnnrl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:44:24.397699 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.397649 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-7cvpl\"" Nov 03 08:44:24.399092 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.399067 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh"] Nov 03 08:44:24.427144 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.427120 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:44:24.427286 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.427168 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2cx\" (UniqueName: \"kubernetes.io/projected/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-kube-api-access-rl2cx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:44:24.427286 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.427191 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:44:24.427286 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.427242 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkkd6\" (UniqueName: \"kubernetes.io/projected/73ac1ca1-e946-4c1b-92e5-688169058619-kube-api-access-bkkd6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:44:24.427286 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.427276 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:44:24.427533 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.427331 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:44:24.427593 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.427563 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:44:24.427641 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.427597 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:44:24.454999 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.454966 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:44:24.469195 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.469158 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx"] Nov 03 08:44:24.474257 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.473919 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:44:24.487042 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.487017 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2cx\" (UniqueName: \"kubernetes.io/projected/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-kube-api-access-rl2cx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:44:24.528213 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.528171 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:44:24.528378 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.528235 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:44:24.528378 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.528256 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:44:24.528378 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.528290 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:44:24.528378 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.528321 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkd6\" (UniqueName: \"kubernetes.io/projected/73ac1ca1-e946-4c1b-92e5-688169058619-kube-api-access-bkkd6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:44:24.528378 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.528363 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wldqm\" (UniqueName: \"kubernetes.io/projected/a2b5e45f-e030-446a-a8ca-2decb19a8384-kube-api-access-wldqm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:44:24.528902 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.528869 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:44:24.529220 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.529182 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:44:24.536637 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.536612 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-w7hsr\"" Nov 03 08:44:24.538532 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.538506 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx"] Nov 03 08:44:24.552081 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.552062 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:44:24.593338 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.593307 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkkd6\" (UniqueName: \"kubernetes.io/projected/73ac1ca1-e946-4c1b-92e5-688169058619-kube-api-access-bkkd6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:44:24.629370 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.629335 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:44:24.629535 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.629397 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:44:24.629535 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.629468 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wldqm\" (UniqueName: \"kubernetes.io/projected/a2b5e45f-e030-446a-a8ca-2decb19a8384-kube-api-access-wldqm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:44:24.629806 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.629782 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:44:24.629852 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.629788 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:44:24.651411 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.651346 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:44:24.690204 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.690168 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wldqm\" (UniqueName: \"kubernetes.io/projected/a2b5e45f-e030-446a-a8ca-2decb19a8384-kube-api-access-wldqm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:44:24.802549 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.802509 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:44:24.836305 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.836160 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf"] Nov 03 08:44:24.839955 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:44:24.839930 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc99055_280d_4def_833b_b73c4e1a8b8a.slice/crio-8096c19526b74154d402b5df1fc6ada78c316995c2a4f8f4c98caf6e481a4f2a WatchSource:0}: Error finding container 8096c19526b74154d402b5df1fc6ada78c316995c2a4f8f4c98caf6e481a4f2a: Status 404 returned error can't find the container with id 8096c19526b74154d402b5df1fc6ada78c316995c2a4f8f4c98caf6e481a4f2a Nov 03 08:44:24.940503 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:24.940417 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns"] Nov 03 08:44:24.944135 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:44:24.944103 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9af3f54_4abd_49f5_a83b_cd3c80ab19b3.slice/crio-6da7b5293c4b46f9d9cfadca422375a2ec54f1b876be5c85d4edf687e2227d3b WatchSource:0}: Error finding container 6da7b5293c4b46f9d9cfadca422375a2ec54f1b876be5c85d4edf687e2227d3b: Status 404 returned error can't find the container with id 6da7b5293c4b46f9d9cfadca422375a2ec54f1b876be5c85d4edf687e2227d3b Nov 03 08:44:25.024795 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:25.024773 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh"] Nov 03 08:44:25.027086 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:44:25.027060 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ac1ca1_e946_4c1b_92e5_688169058619.slice/crio-8afe69e396ecc56f38834892b30af60499d0719f7bf2abcc8ab2cf4349b450df WatchSource:0}: Error finding container 8afe69e396ecc56f38834892b30af60499d0719f7bf2abcc8ab2cf4349b450df: Status 404 returned error can't find the container with id 8afe69e396ecc56f38834892b30af60499d0719f7bf2abcc8ab2cf4349b450df Nov 03 08:44:25.178970 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:25.178940 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx"] Nov 03 08:44:25.181567 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:44:25.181518 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b5e45f_e030_446a_a8ca_2decb19a8384.slice/crio-ec94d70e4e72e8760d0d2a19e946aa42ad78a0467a8b7497c9c2b665ef404d35 WatchSource:0}: Error finding container ec94d70e4e72e8760d0d2a19e946aa42ad78a0467a8b7497c9c2b665ef404d35: Status 404 returned error can't find the container with id ec94d70e4e72e8760d0d2a19e946aa42ad78a0467a8b7497c9c2b665ef404d35 Nov 03 08:44:25.503943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:25.503902 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" event={"ID":"a2b5e45f-e030-446a-a8ca-2decb19a8384","Type":"ContainerStarted","Data":"ec94d70e4e72e8760d0d2a19e946aa42ad78a0467a8b7497c9c2b665ef404d35"} Nov 03 08:44:25.505120 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:25.505095 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" event={"ID":"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3","Type":"ContainerStarted","Data":"6da7b5293c4b46f9d9cfadca422375a2ec54f1b876be5c85d4edf687e2227d3b"} Nov 03 08:44:25.506253 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:25.506234 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" event={"ID":"efc99055-280d-4def-833b-b73c4e1a8b8a","Type":"ContainerStarted","Data":"8096c19526b74154d402b5df1fc6ada78c316995c2a4f8f4c98caf6e481a4f2a"} Nov 03 08:44:25.507397 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:25.507379 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" event={"ID":"73ac1ca1-e946-4c1b-92e5-688169058619","Type":"ContainerStarted","Data":"8afe69e396ecc56f38834892b30af60499d0719f7bf2abcc8ab2cf4349b450df"} Nov 03 08:44:26.331893 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.331854 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd"] Nov 03 08:44:26.336091 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.336067 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:44:26.441974 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.441932 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-x7xpb\"" Nov 03 08:44:26.444349 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.444318 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd"] Nov 03 08:44:26.446101 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.446072 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:44:26.446228 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.446134 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hxq2\" (UniqueName: \"kubernetes.io/projected/51d3dcff-05ea-4ab2-910b-7a2206274a0a-kube-api-access-5hxq2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:44:26.446228 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.446199 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:44:26.547217 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.547173 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:44:26.547430 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.547236 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hxq2\" (UniqueName: \"kubernetes.io/projected/51d3dcff-05ea-4ab2-910b-7a2206274a0a-kube-api-access-5hxq2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:44:26.547430 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.547289 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:44:26.548338 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.548296 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:44:26.548465 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.548391 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:44:26.617463 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.617305 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hxq2\" (UniqueName: \"kubernetes.io/projected/51d3dcff-05ea-4ab2-910b-7a2206274a0a-kube-api-access-5hxq2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:44:26.657309 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.657029 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:44:26.686246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.686212 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc"] Nov 03 08:44:26.690732 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.690706 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:44:26.754616 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.754570 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-xq6t6\"" Nov 03 08:44:26.755310 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.755284 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:44:26.755425 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.755355 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:44:26.755526 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.755497 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh24t\" (UniqueName: \"kubernetes.io/projected/68e88181-3bd5-4803-b24d-a69734199601-kube-api-access-zh24t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:44:26.757711 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.757685 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc"] Nov 03 08:44:26.856687 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.856653 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:44:26.856910 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.856720 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh24t\" (UniqueName: \"kubernetes.io/projected/68e88181-3bd5-4803-b24d-a69734199601-kube-api-access-zh24t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:44:26.856910 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.856778 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:44:26.857151 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.857109 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:44:26.857331 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.857290 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:44:26.920131 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:26.920016 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh24t\" (UniqueName: \"kubernetes.io/projected/68e88181-3bd5-4803-b24d-a69734199601-kube-api-access-zh24t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:44:27.016106 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:27.015769 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:44:27.063238 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:27.063173 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd"] Nov 03 08:44:27.067624 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:44:27.067590 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d3dcff_05ea_4ab2_910b_7a2206274a0a.slice/crio-e798fedee070f201d778f34953621e6048b97238bd888ce6256eb8bff3a2d2b7 WatchSource:0}: Error finding container e798fedee070f201d778f34953621e6048b97238bd888ce6256eb8bff3a2d2b7: Status 404 returned error can't find the container with id e798fedee070f201d778f34953621e6048b97238bd888ce6256eb8bff3a2d2b7 Nov 03 08:44:27.234376 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:27.234190 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 08:44:27.237101 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:27.237059 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 08:44:27.429720 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:27.429688 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc"] Nov 03 08:44:27.437661 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:44:27.437626 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e88181_3bd5_4803_b24d_a69734199601.slice/crio-3557cc8b8b634a0f9b920bf769ef82dd0e7ad4110796ebae8070a8d4930b2ea8 WatchSource:0}: Error finding container 3557cc8b8b634a0f9b920bf769ef82dd0e7ad4110796ebae8070a8d4930b2ea8: Status 404 returned error can't find the container with id 3557cc8b8b634a0f9b920bf769ef82dd0e7ad4110796ebae8070a8d4930b2ea8 Nov 03 08:44:27.522792 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:27.522647 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" event={"ID":"68e88181-3bd5-4803-b24d-a69734199601","Type":"ContainerStarted","Data":"3557cc8b8b634a0f9b920bf769ef82dd0e7ad4110796ebae8070a8d4930b2ea8"} Nov 03 08:44:27.525187 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:27.525155 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" event={"ID":"51d3dcff-05ea-4ab2-910b-7a2206274a0a","Type":"ContainerStarted","Data":"e798fedee070f201d778f34953621e6048b97238bd888ce6256eb8bff3a2d2b7"} Nov 03 08:44:29.943836 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:29.943796 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh"] Nov 03 08:44:29.947850 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:29.947818 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:44:30.010738 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.010705 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh"] Nov 03 08:44:30.019689 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.016105 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-tfctb\"" Nov 03 08:44:30.090821 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.090460 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvr6\" (UniqueName: \"kubernetes.io/projected/8f9ac048-a91e-4ba9-8299-4f241d0734eb-kube-api-access-9jvr6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:44:30.090821 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.090553 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:44:30.090821 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.090601 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:44:30.192747 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.191428 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:44:30.192747 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.191523 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:44:30.192747 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.191624 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvr6\" (UniqueName: \"kubernetes.io/projected/8f9ac048-a91e-4ba9-8299-4f241d0734eb-kube-api-access-9jvr6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:44:30.192747 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.192410 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:44:30.192747 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.192691 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:44:30.257511 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.252287 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvr6\" (UniqueName: \"kubernetes.io/projected/8f9ac048-a91e-4ba9-8299-4f241d0734eb-kube-api-access-9jvr6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:44:30.273096 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.273032 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:44:30.722726 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.722692 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh"] Nov 03 08:44:30.733169 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.733149 5015 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:44:30.912525 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.912446 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt"] Nov 03 08:44:30.919811 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.919781 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:30.996225 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.996140 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt"] Nov 03 08:44:30.997278 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:30.996305 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 08:44:31.000888 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.000859 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.001022 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.000916 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.001022 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.000956 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f961ac-69cf-407a-af9b-07611b20726c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.001022 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.000989 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.001218 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.001023 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.001218 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.001068 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnjlp\" (UniqueName: \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-kube-api-access-fnjlp\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.001218 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.001121 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.001218 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.001146 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f961ac-69cf-407a-af9b-07611b20726c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.001218 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.001177 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.025189 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.025152 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pqxg4\"" Nov 03 08:44:31.102449 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.102417 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f961ac-69cf-407a-af9b-07611b20726c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.102654 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.102458 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.102654 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.102519 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.102654 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.102569 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnjlp\" (UniqueName: \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-kube-api-access-fnjlp\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.102654 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.102626 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.102654 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.102651 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f961ac-69cf-407a-af9b-07611b20726c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.103001 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.102687 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.103001 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.102740 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.103001 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.102771 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.103001 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.102945 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.103208 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.103123 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.103273 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.103220 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.103727 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.103367 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f961ac-69cf-407a-af9b-07611b20726c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.103727 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.103681 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.111323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.111297 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f961ac-69cf-407a-af9b-07611b20726c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.112857 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.112810 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.165444 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.165368 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.166260 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.166192 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnjlp\" (UniqueName: \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-kube-api-access-fnjlp\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c89vt\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.209885 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.209848 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb"] Nov 03 08:44:31.216426 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.216401 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:44:31.241744 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.241554 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:31.250047 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.248912 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7"] Nov 03 08:44:31.253733 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.253710 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v"] Nov 03 08:44:31.255923 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.255895 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.259797 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.259664 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:44:31.268939 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.268729 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb"] Nov 03 08:44:31.304672 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.304638 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d498h\" (UniqueName: \"kubernetes.io/projected/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-kube-api-access-d498h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:44:31.304881 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.304704 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:44:31.304881 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.304749 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:44:31.311621 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.311597 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-hq8g5\"" Nov 03 08:44:31.330140 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.330103 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v"] Nov 03 08:44:31.330271 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.330165 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7"] Nov 03 08:44:31.346317 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.346286 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh"] Nov 03 08:44:31.354659 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.354629 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:44:31.407204 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.407154 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:44:31.407369 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.407236 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:44:31.408243 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.408222 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:44:31.410710 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.408384 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:44:31.410710 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.410633 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh"] Nov 03 08:44:31.410995 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.410965 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.411112 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411032 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.411112 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411067 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.411227 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411122 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.411227 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411162 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv57w\" (UniqueName: \"kubernetes.io/projected/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-kube-api-access-jv57w\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.411227 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411184 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:44:31.411227 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411223 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d498h\" (UniqueName: \"kubernetes.io/projected/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-kube-api-access-d498h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:44:31.411431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411251 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.411431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411277 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.411431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411306 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:44:31.411431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411375 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbpgj\" (UniqueName: \"kubernetes.io/projected/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-kube-api-access-vbpgj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:44:31.411431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411403 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.411984 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.411425 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.475559 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.473247 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d498h\" (UniqueName: \"kubernetes.io/projected/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-kube-api-access-d498h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:44:31.512113 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512025 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv57w\" (UniqueName: \"kubernetes.io/projected/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-kube-api-access-jv57w\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.512113 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512076 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:44:31.512113 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512107 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:44:31.512398 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512177 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.512398 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512204 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.512398 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512228 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:44:31.512398 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512294 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbpgj\" (UniqueName: \"kubernetes.io/projected/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-kube-api-access-vbpgj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:44:31.512398 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512321 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.512398 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512343 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.512398 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512380 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:44:31.512832 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512433 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.512832 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512489 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.512832 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512524 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.512832 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512581 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.512832 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.512607 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68zfq\" (UniqueName: \"kubernetes.io/projected/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-kube-api-access-68zfq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:44:31.513309 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.513283 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:44:31.513672 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.513652 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.514132 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.514110 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:44:31.515042 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.514819 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.515042 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.514991 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.515553 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.515441 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.515956 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.515899 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.519517 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.519448 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.520326 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.520286 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.539558 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.539384 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:44:31.565585 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.565551 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" event={"ID":"8f9ac048-a91e-4ba9-8299-4f241d0734eb","Type":"ContainerStarted","Data":"81e9c97b30401a070f21be1792bd3f88862693f7071eaf75d460277557861be4"} Nov 03 08:44:31.577899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.577866 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbpgj\" (UniqueName: \"kubernetes.io/projected/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-kube-api-access-vbpgj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:44:31.578706 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.578679 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.593206 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.592691 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv57w\" (UniqueName: \"kubernetes.io/projected/f9d5e9a1-bee4-4750-90bf-892c58ff29a5-kube-api-access-jv57w\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7\" (UID: \"f9d5e9a1-bee4-4750-90bf-892c58ff29a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.597033 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.596920 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:31.613843 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.613633 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:44:31.613843 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.613807 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:44:31.614115 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.613890 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68zfq\" (UniqueName: \"kubernetes.io/projected/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-kube-api-access-68zfq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:44:31.614115 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.613931 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:44:31.614335 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.614312 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:44:31.614637 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.614616 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:44:31.671131 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.670819 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt"] Nov 03 08:44:31.672156 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:44:31.671657 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f961ac_69cf_407a_af9b_07611b20726c.slice/crio-fd76f7f8f59b1c3d4f7845b1d1f7c2a5f8091f6034bde1a9901bcfe24259683e WatchSource:0}: Error finding container fd76f7f8f59b1c3d4f7845b1d1f7c2a5f8091f6034bde1a9901bcfe24259683e: Status 404 returned error can't find the container with id fd76f7f8f59b1c3d4f7845b1d1f7c2a5f8091f6034bde1a9901bcfe24259683e Nov 03 08:44:31.685847 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.683382 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68zfq\" (UniqueName: \"kubernetes.io/projected/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-kube-api-access-68zfq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:44:31.685847 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.683958 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:44:31.964914 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:31.964390 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb"] Nov 03 08:44:31.978234 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:44:31.978188 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92f1ea2_322b_4fde_9525_2a16df2bd6b7.slice/crio-1e62a218331c31da8058faf7864f100d0726d087ac744e7809df0f75bc8c8ad0 WatchSource:0}: Error finding container 1e62a218331c31da8058faf7864f100d0726d087ac744e7809df0f75bc8c8ad0: Status 404 returned error can't find the container with id 1e62a218331c31da8058faf7864f100d0726d087ac744e7809df0f75bc8c8ad0 Nov 03 08:44:32.028111 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:32.028075 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7"] Nov 03 08:44:32.033156 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:44:32.033106 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d5e9a1_bee4_4750_90bf_892c58ff29a5.slice/crio-5af217fd29bbe77a2eaaae7b1325ce449d965854e0007b7c788a5d0876fade76 WatchSource:0}: Error finding container 5af217fd29bbe77a2eaaae7b1325ce449d965854e0007b7c788a5d0876fade76: Status 404 returned error can't find the container with id 5af217fd29bbe77a2eaaae7b1325ce449d965854e0007b7c788a5d0876fade76 Nov 03 08:44:32.042136 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:32.040756 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v"] Nov 03 08:44:32.045221 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:44:32.045196 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac41e6f_2f9d_4d2c_af89_223460ffb22f.slice/crio-09b4becfa552a880a00699b2505f7f3953bef47959e5d221dce83fa343c3090c WatchSource:0}: Error finding container 09b4becfa552a880a00699b2505f7f3953bef47959e5d221dce83fa343c3090c: Status 404 returned error can't find the container with id 09b4becfa552a880a00699b2505f7f3953bef47959e5d221dce83fa343c3090c Nov 03 08:44:32.136133 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:32.136088 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh"] Nov 03 08:44:32.145435 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:44:32.145401 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b53b3f3_51f9_4b7e_ab25_efe7c4938089.slice/crio-9de3ad68e3c4a45c8f821bfafef3fc88fce1190e80932bc5b76372ea03ecdec1 WatchSource:0}: Error finding container 9de3ad68e3c4a45c8f821bfafef3fc88fce1190e80932bc5b76372ea03ecdec1: Status 404 returned error can't find the container with id 9de3ad68e3c4a45c8f821bfafef3fc88fce1190e80932bc5b76372ea03ecdec1 Nov 03 08:44:32.578632 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:32.578525 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" event={"ID":"66f961ac-69cf-407a-af9b-07611b20726c","Type":"ContainerStarted","Data":"fd76f7f8f59b1c3d4f7845b1d1f7c2a5f8091f6034bde1a9901bcfe24259683e"} Nov 03 08:44:32.583888 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:32.583724 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" event={"ID":"5b53b3f3-51f9-4b7e-ab25-efe7c4938089","Type":"ContainerStarted","Data":"9de3ad68e3c4a45c8f821bfafef3fc88fce1190e80932bc5b76372ea03ecdec1"} Nov 03 08:44:32.592270 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:32.592146 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" event={"ID":"5ac41e6f-2f9d-4d2c-af89-223460ffb22f","Type":"ContainerStarted","Data":"09b4becfa552a880a00699b2505f7f3953bef47959e5d221dce83fa343c3090c"} Nov 03 08:44:32.594418 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:32.594340 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" event={"ID":"f9d5e9a1-bee4-4750-90bf-892c58ff29a5","Type":"ContainerStarted","Data":"5af217fd29bbe77a2eaaae7b1325ce449d965854e0007b7c788a5d0876fade76"} Nov 03 08:44:32.596345 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:32.596320 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" event={"ID":"a92f1ea2-322b-4fde-9525-2a16df2bd6b7","Type":"ContainerStarted","Data":"1e62a218331c31da8058faf7864f100d0726d087ac744e7809df0f75bc8c8ad0"} Nov 03 08:44:44.143272 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.143234 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:44:44.143748 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.143314 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:44:44.143748 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.143344 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:44:44.558467 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.557988 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:44:44.558467 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.558084 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:44:44.558467 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.558129 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:44:44.735792 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.734917 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" event={"ID":"f9d5e9a1-bee4-4750-90bf-892c58ff29a5","Type":"ContainerStarted","Data":"e12982d38a9b2f4fcc08b8fcb461290c8f59f6473c49e2f01eb8ed48728d4d65"} Nov 03 08:44:44.754739 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.745445 5015 generic.go:356] "Generic (PLEG): container finished" podID="e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" containerID="1fd6770f08768629b29bd495b05149f063e05a789a4bf3e97232ecb5411bf527" exitCode=0 Nov 03 08:44:44.754739 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.745587 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" event={"ID":"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3","Type":"ContainerDied","Data":"1fd6770f08768629b29bd495b05149f063e05a789a4bf3e97232ecb5411bf527"} Nov 03 08:44:44.754739 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.751169 5015 generic.go:356] "Generic (PLEG): container finished" podID="efc99055-280d-4def-833b-b73c4e1a8b8a" containerID="e4eef9ba2c18a7b2e57c742b3e9f5095d18711e39d8b57c4080be26fc47f003d" exitCode=0 Nov 03 08:44:44.754739 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.751269 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" event={"ID":"efc99055-280d-4def-833b-b73c4e1a8b8a","Type":"ContainerDied","Data":"e4eef9ba2c18a7b2e57c742b3e9f5095d18711e39d8b57c4080be26fc47f003d"} Nov 03 08:44:44.766119 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.766057 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" event={"ID":"8f9ac048-a91e-4ba9-8299-4f241d0734eb","Type":"ContainerStarted","Data":"23ebf2f7eccd51b4952026af4d9d0c2ca7cb1ca490826f85311b4a6873d011f3"} Nov 03 08:44:44.770591 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.770036 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" event={"ID":"66f961ac-69cf-407a-af9b-07611b20726c","Type":"ContainerStarted","Data":"ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781"} Nov 03 08:44:44.775180 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.774741 5015 generic.go:356] "Generic (PLEG): container finished" podID="73ac1ca1-e946-4c1b-92e5-688169058619" containerID="6cd6b4195352c11003627995eaef02ccbd3067440ae44da8333b76254f0769b6" exitCode=0 Nov 03 08:44:44.775180 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.774850 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" event={"ID":"73ac1ca1-e946-4c1b-92e5-688169058619","Type":"ContainerDied","Data":"6cd6b4195352c11003627995eaef02ccbd3067440ae44da8333b76254f0769b6"} Nov 03 08:44:44.887803 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:44.887738 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" podStartSLOduration=1.788225258 podStartE2EDuration="13.887720585s" totalImagesPullingTime="12.099495327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:44:44.884993932 +0000 UTC m=+317.809936168" watchObservedRunningTime="2026-11-03 08:44:44.887720585 +0000 UTC m=+317.812662800" Nov 03 08:44:45.242729 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.242630 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:45.248451 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.248425 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:45.363780 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.363669 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" podStartSLOduration=2.512042802 podStartE2EDuration="15.363647998s" totalImagesPullingTime="12.851605196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:44:45.362371319 +0000 UTC m=+318.287313565" watchObservedRunningTime="2026-11-03 08:44:45.363647998 +0000 UTC m=+318.288590217" Nov 03 08:44:45.597236 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.597153 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:45.603375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.603349 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:45.782687 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.782639 5015 generic.go:356] "Generic (PLEG): container finished" podID="a2b5e45f-e030-446a-a8ca-2decb19a8384" containerID="6c8160a8584cc99c62e7a8e713490fb7692ed8af9914ed1515b04cb9dbb62286" exitCode=0 Nov 03 08:44:45.782870 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.782725 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" event={"ID":"a2b5e45f-e030-446a-a8ca-2decb19a8384","Type":"ContainerDied","Data":"6c8160a8584cc99c62e7a8e713490fb7692ed8af9914ed1515b04cb9dbb62286"} Nov 03 08:44:45.784812 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.784787 5015 generic.go:356] "Generic (PLEG): container finished" podID="a92f1ea2-322b-4fde-9525-2a16df2bd6b7" containerID="9e52cf899ef061ccf52ca8dc305b207ff099547d69bbb7cd2d06e70bb50c63f9" exitCode=0 Nov 03 08:44:45.784910 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.784898 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" event={"ID":"a92f1ea2-322b-4fde-9525-2a16df2bd6b7","Type":"ContainerDied","Data":"9e52cf899ef061ccf52ca8dc305b207ff099547d69bbb7cd2d06e70bb50c63f9"} Nov 03 08:44:45.787055 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.787012 5015 generic.go:356] "Generic (PLEG): container finished" podID="8f9ac048-a91e-4ba9-8299-4f241d0734eb" containerID="23ebf2f7eccd51b4952026af4d9d0c2ca7cb1ca490826f85311b4a6873d011f3" exitCode=0 Nov 03 08:44:45.787136 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.787112 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" event={"ID":"8f9ac048-a91e-4ba9-8299-4f241d0734eb","Type":"ContainerDied","Data":"23ebf2f7eccd51b4952026af4d9d0c2ca7cb1ca490826f85311b4a6873d011f3"} Nov 03 08:44:45.789062 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.788850 5015 generic.go:356] "Generic (PLEG): container finished" podID="5b53b3f3-51f9-4b7e-ab25-efe7c4938089" containerID="5a3c1f245329b0e1f2c4b489410cad85538110dc66f4e7a3ed1be6814a4aab42" exitCode=0 Nov 03 08:44:45.789062 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.788927 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" event={"ID":"5b53b3f3-51f9-4b7e-ab25-efe7c4938089","Type":"ContainerDied","Data":"5a3c1f245329b0e1f2c4b489410cad85538110dc66f4e7a3ed1be6814a4aab42"} Nov 03 08:44:45.791388 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.791363 5015 generic.go:356] "Generic (PLEG): container finished" podID="5ac41e6f-2f9d-4d2c-af89-223460ffb22f" containerID="aa035f5e1fbf719fe208a5cd965d082898212036e7c846ad31d725141b82a124" exitCode=0 Nov 03 08:44:45.791504 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.791438 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" event={"ID":"5ac41e6f-2f9d-4d2c-af89-223460ffb22f","Type":"ContainerDied","Data":"aa035f5e1fbf719fe208a5cd965d082898212036e7c846ad31d725141b82a124"} Nov 03 08:44:45.791988 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.791971 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:45.792224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.792012 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:45.793112 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.793092 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7" Nov 03 08:44:45.793282 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:45.793267 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:44:46.797760 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:46.797722 5015 generic.go:356] "Generic (PLEG): container finished" podID="68e88181-3bd5-4803-b24d-a69734199601" containerID="ccaa42873f13b91d47a0875f0d464e69b349b67c38574970ab50e4711f02cead" exitCode=0 Nov 03 08:44:46.798228 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:46.797817 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" event={"ID":"68e88181-3bd5-4803-b24d-a69734199601","Type":"ContainerDied","Data":"ccaa42873f13b91d47a0875f0d464e69b349b67c38574970ab50e4711f02cead"} Nov 03 08:44:46.799516 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:46.799392 5015 generic.go:356] "Generic (PLEG): container finished" podID="51d3dcff-05ea-4ab2-910b-7a2206274a0a" containerID="fcfc5ca70814d41cdb8454f6c3900a24ad16ba1ec2dae862137dffca2255e915" exitCode=0 Nov 03 08:44:46.799516 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:46.799499 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" event={"ID":"51d3dcff-05ea-4ab2-910b-7a2206274a0a","Type":"ContainerDied","Data":"fcfc5ca70814d41cdb8454f6c3900a24ad16ba1ec2dae862137dffca2255e915"} Nov 03 08:44:46.801551 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:46.801522 5015 generic.go:356] "Generic (PLEG): container finished" podID="e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" containerID="2eecce3096dd964821fb1dda211728b8f62ee8a5834c47483468d76f6a029310" exitCode=0 Nov 03 08:44:46.801657 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:46.801602 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" event={"ID":"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3","Type":"ContainerDied","Data":"2eecce3096dd964821fb1dda211728b8f62ee8a5834c47483468d76f6a029310"} Nov 03 08:44:46.901852 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:44:46.901818 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:44:47.811184 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:47.811147 5015 generic.go:356] "Generic (PLEG): container finished" podID="a2b5e45f-e030-446a-a8ca-2decb19a8384" containerID="4bfed65ba60d0e52817bac93d9d7be193d6a05c1ee667ea93ae76397a35d7eb9" exitCode=0 Nov 03 08:44:47.811646 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:47.811247 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" event={"ID":"a2b5e45f-e030-446a-a8ca-2decb19a8384","Type":"ContainerDied","Data":"4bfed65ba60d0e52817bac93d9d7be193d6a05c1ee667ea93ae76397a35d7eb9"} Nov 03 08:44:47.814445 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:47.814415 5015 generic.go:356] "Generic (PLEG): container finished" podID="a92f1ea2-322b-4fde-9525-2a16df2bd6b7" containerID="be20cb9f5f500851bc998e77f627b34d406e9382eb4562d3487f1a2b9e814559" exitCode=0 Nov 03 08:44:47.815659 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:47.814620 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" event={"ID":"a92f1ea2-322b-4fde-9525-2a16df2bd6b7","Type":"ContainerDied","Data":"be20cb9f5f500851bc998e77f627b34d406e9382eb4562d3487f1a2b9e814559"} Nov 03 08:44:47.817797 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:47.817767 5015 generic.go:356] "Generic (PLEG): container finished" podID="5b53b3f3-51f9-4b7e-ab25-efe7c4938089" containerID="cd1b3b5f827c0c010024027184b586ea26e38412b06fce5db13d3feced2f8088" exitCode=0 Nov 03 08:44:47.817884 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:47.817825 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" event={"ID":"5b53b3f3-51f9-4b7e-ab25-efe7c4938089","Type":"ContainerDied","Data":"cd1b3b5f827c0c010024027184b586ea26e38412b06fce5db13d3feced2f8088"} Nov 03 08:44:48.832019 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:48.831976 5015 generic.go:356] "Generic (PLEG): container finished" podID="68e88181-3bd5-4803-b24d-a69734199601" containerID="801ad4de0b03e30fdb08afb2ff03c0a09f9edcccdeebfbf8abb2f02ece004ece" exitCode=0 Nov 03 08:44:48.832441 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:48.832171 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" event={"ID":"68e88181-3bd5-4803-b24d-a69734199601","Type":"ContainerDied","Data":"801ad4de0b03e30fdb08afb2ff03c0a09f9edcccdeebfbf8abb2f02ece004ece"} Nov 03 08:44:48.838848 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:48.838816 5015 generic.go:356] "Generic (PLEG): container finished" podID="efc99055-280d-4def-833b-b73c4e1a8b8a" containerID="f1bd5e80ae405f45f3aae417699209948988493721f44f5e15898354707327e5" exitCode=0 Nov 03 08:44:48.838948 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:48.838891 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" event={"ID":"efc99055-280d-4def-833b-b73c4e1a8b8a","Type":"ContainerDied","Data":"f1bd5e80ae405f45f3aae417699209948988493721f44f5e15898354707327e5"} Nov 03 08:44:49.859756 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:49.859688 5015 generic.go:356] "Generic (PLEG): container finished" podID="8f9ac048-a91e-4ba9-8299-4f241d0734eb" containerID="c864cfc89d53d786315bf3e3fef71d8fb7bbde1f39cbe56ff72a677c78d82993" exitCode=0 Nov 03 08:44:49.860243 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:49.859804 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" event={"ID":"8f9ac048-a91e-4ba9-8299-4f241d0734eb","Type":"ContainerDied","Data":"c864cfc89d53d786315bf3e3fef71d8fb7bbde1f39cbe56ff72a677c78d82993"} Nov 03 08:44:49.867985 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:49.867952 5015 generic.go:356] "Generic (PLEG): container finished" podID="73ac1ca1-e946-4c1b-92e5-688169058619" containerID="549a56d5b750fdf5e00c8a13e844ed76b54d78da15b0b3085afe11184fb8c8e6" exitCode=0 Nov 03 08:44:49.868111 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:49.868018 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" event={"ID":"73ac1ca1-e946-4c1b-92e5-688169058619","Type":"ContainerDied","Data":"549a56d5b750fdf5e00c8a13e844ed76b54d78da15b0b3085afe11184fb8c8e6"} Nov 03 08:44:53.945413 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:53.945371 5015 generic.go:356] "Generic (PLEG): container finished" podID="51d3dcff-05ea-4ab2-910b-7a2206274a0a" containerID="9c2b3ee298e2608177ebc51830f0d48ad05ee54a58f80d4f549368db9988ac47" exitCode=0 Nov 03 08:44:53.945951 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:53.945502 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" event={"ID":"51d3dcff-05ea-4ab2-910b-7a2206274a0a","Type":"ContainerDied","Data":"9c2b3ee298e2608177ebc51830f0d48ad05ee54a58f80d4f549368db9988ac47"} Nov 03 08:44:53.956370 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:53.956332 5015 generic.go:356] "Generic (PLEG): container finished" podID="5ac41e6f-2f9d-4d2c-af89-223460ffb22f" containerID="a4fd47a32aad3b85e5bb5401ec4712888f04f005804cc4f659193233845a6288" exitCode=0 Nov 03 08:44:53.956535 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:44:53.956470 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" event={"ID":"5ac41e6f-2f9d-4d2c-af89-223460ffb22f","Type":"ContainerDied","Data":"a4fd47a32aad3b85e5bb5401ec4712888f04f005804cc4f659193233845a6288"} Nov 03 08:45:07.104912 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:07.104845 5015 generic.go:356] "Generic (PLEG): container finished" podID="e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" containerID="470ceb4892e92cc8842c3ad8e8b2b3a2ed9a7464c9a9e18894287710d48b9c72" exitCode=0 Nov 03 08:45:07.105432 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:07.105030 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" event={"ID":"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3","Type":"ContainerDied","Data":"470ceb4892e92cc8842c3ad8e8b2b3a2ed9a7464c9a9e18894287710d48b9c72"} Nov 03 08:45:07.109087 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:07.109055 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" event={"ID":"51d3dcff-05ea-4ab2-910b-7a2206274a0a","Type":"ContainerStarted","Data":"79dae458155e606a6cd7fb38488998a9fd8fa2c898c633570eff23f2f7f6fb63"} Nov 03 08:45:07.426218 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:07.426100 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" podStartSLOduration=3.666122869 podStartE2EDuration="41.426079399s" totalImagesPullingTime="37.75995653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:07.425419614 +0000 UTC m=+340.350361833" watchObservedRunningTime="2026-11-03 08:45:07.426079399 +0000 UTC m=+340.351021617" Nov 03 08:45:08.121893 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.121235 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" event={"ID":"a2b5e45f-e030-446a-a8ca-2decb19a8384","Type":"ContainerStarted","Data":"07d9dfcad9226985f32541185c5d4082b506432085c53bc162666305b20051c3"} Nov 03 08:45:08.125272 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.125228 5015 generic.go:356] "Generic (PLEG): container finished" podID="51d3dcff-05ea-4ab2-910b-7a2206274a0a" containerID="79dae458155e606a6cd7fb38488998a9fd8fa2c898c633570eff23f2f7f6fb63" exitCode=0 Nov 03 08:45:08.125432 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.125318 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" event={"ID":"51d3dcff-05ea-4ab2-910b-7a2206274a0a","Type":"ContainerDied","Data":"79dae458155e606a6cd7fb38488998a9fd8fa2c898c633570eff23f2f7f6fb63"} Nov 03 08:45:08.131338 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.130898 5015 generic.go:356] "Generic (PLEG): container finished" podID="5ac41e6f-2f9d-4d2c-af89-223460ffb22f" containerID="42d6de4c7c25c2783aa0e662c2b2d910429707b83a9084a5084dd8d4bdb7ad2b" exitCode=0 Nov 03 08:45:08.131338 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.131287 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" event={"ID":"5ac41e6f-2f9d-4d2c-af89-223460ffb22f","Type":"ContainerDied","Data":"42d6de4c7c25c2783aa0e662c2b2d910429707b83a9084a5084dd8d4bdb7ad2b"} Nov 03 08:45:08.339365 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.339257 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:45:08.360710 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.360656 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" podStartSLOduration=2.999983172 podStartE2EDuration="44.360638718s" totalImagesPullingTime="41.360655546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:08.239332434 +0000 UTC m=+341.164274653" watchObservedRunningTime="2026-11-03 08:45:08.360638718 +0000 UTC m=+341.285580971" Nov 03 08:45:08.436109 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.436027 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-kube-api-access-rl2cx\" (UniqueName: \"kubernetes.io/projected/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-kube-api-access-rl2cx\") pod \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " Nov 03 08:45:08.436274 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.436139 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-bundle\") pod \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " Nov 03 08:45:08.436274 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.436178 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-util\" (UniqueName: \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-util\") pod \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\" (UID: \"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3\") " Nov 03 08:45:08.443618 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.443575 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-bundle" pod "e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" (UID: "e9af3f54-4abd-49f5-a83b-cd3c80ab19b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:08.444947 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.444922 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-util" pod "e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" (UID: "e9af3f54-4abd-49f5-a83b-cd3c80ab19b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:08.457177 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.457134 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-kube-api-access-rl2cx" pod "e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" (UID: "e9af3f54-4abd-49f5-a83b-cd3c80ab19b3"). InnerVolumeSpecName "kube-api-access-rl2cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:45:08.536966 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.536941 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2cx\" (UniqueName: \"kubernetes.io/projected/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-kube-api-access-rl2cx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:08.537099 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.536970 5015 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:08.537099 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:08.536984 5015 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9af3f54-4abd-49f5-a83b-cd3c80ab19b3-util\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:09.145362 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.145258 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" event={"ID":"efc99055-280d-4def-833b-b73c4e1a8b8a","Type":"ContainerStarted","Data":"a50c6c730082cd3c3dfd9d465ed853dea6bed5b585c5cd59216509849015d0f3"} Nov 03 08:45:09.148885 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.148846 5015 generic.go:356] "Generic (PLEG): container finished" podID="5b53b3f3-51f9-4b7e-ab25-efe7c4938089" containerID="04d4e1bfd873f4bf42a9d07b3164eb1aa1324e3ad6898e35a7450f878009792c" exitCode=0 Nov 03 08:45:09.149147 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.149125 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" event={"ID":"5b53b3f3-51f9-4b7e-ab25-efe7c4938089","Type":"ContainerDied","Data":"04d4e1bfd873f4bf42a9d07b3164eb1aa1324e3ad6898e35a7450f878009792c"} Nov 03 08:45:09.153159 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.153128 5015 generic.go:356] "Generic (PLEG): container finished" podID="a2b5e45f-e030-446a-a8ca-2decb19a8384" containerID="07d9dfcad9226985f32541185c5d4082b506432085c53bc162666305b20051c3" exitCode=0 Nov 03 08:45:09.153334 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.153314 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" event={"ID":"a2b5e45f-e030-446a-a8ca-2decb19a8384","Type":"ContainerDied","Data":"07d9dfcad9226985f32541185c5d4082b506432085c53bc162666305b20051c3"} Nov 03 08:45:09.157283 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.157257 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" event={"ID":"e9af3f54-4abd-49f5-a83b-cd3c80ab19b3","Type":"ContainerDied","Data":"6da7b5293c4b46f9d9cfadca422375a2ec54f1b876be5c85d4edf687e2227d3b"} Nov 03 08:45:09.157400 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.157300 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6da7b5293c4b46f9d9cfadca422375a2ec54f1b876be5c85d4edf687e2227d3b" Nov 03 08:45:09.157400 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.157264 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztgns" Nov 03 08:45:09.169793 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.169075 5015 generic.go:356] "Generic (PLEG): container finished" podID="a92f1ea2-322b-4fde-9525-2a16df2bd6b7" containerID="520235324a67349ffd6779520f620f5ddfb4b5a6e5b8e7af4753df46c8131aa2" exitCode=0 Nov 03 08:45:09.169793 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.169445 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" event={"ID":"a92f1ea2-322b-4fde-9525-2a16df2bd6b7","Type":"ContainerDied","Data":"520235324a67349ffd6779520f620f5ddfb4b5a6e5b8e7af4753df46c8131aa2"} Nov 03 08:45:09.360705 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.360663 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:45:09.389276 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.389249 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:45:09.394758 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.394702 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" podStartSLOduration=3.235398959 podStartE2EDuration="45.394681673s" totalImagesPullingTime="42.159282714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:09.265823323 +0000 UTC m=+342.190765540" watchObservedRunningTime="2026-11-03 08:45:09.394681673 +0000 UTC m=+342.319623891" Nov 03 08:45:09.445936 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.445901 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-util\" (UniqueName: \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-util\") pod \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " Nov 03 08:45:09.445936 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.445941 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d3dcff-05ea-4ab2-910b-7a2206274a0a-kube-api-access-5hxq2\" (UniqueName: \"kubernetes.io/projected/51d3dcff-05ea-4ab2-910b-7a2206274a0a-kube-api-access-5hxq2\") pod \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " Nov 03 08:45:09.446193 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.445978 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-kube-api-access-vbpgj\" (UniqueName: \"kubernetes.io/projected/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-kube-api-access-vbpgj\") pod \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " Nov 03 08:45:09.446193 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.446049 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-bundle\") pod \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " Nov 03 08:45:09.446193 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.446088 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-bundle\") pod \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\" (UID: \"5ac41e6f-2f9d-4d2c-af89-223460ffb22f\") " Nov 03 08:45:09.446193 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.446135 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-util\" (UniqueName: \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-util\") pod \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\" (UID: \"51d3dcff-05ea-4ab2-910b-7a2206274a0a\") " Nov 03 08:45:09.446882 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.446848 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-bundle" pod "51d3dcff-05ea-4ab2-910b-7a2206274a0a" (UID: "51d3dcff-05ea-4ab2-910b-7a2206274a0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:09.447192 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.447145 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-bundle" pod "5ac41e6f-2f9d-4d2c-af89-223460ffb22f" (UID: "5ac41e6f-2f9d-4d2c-af89-223460ffb22f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:09.449508 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.449458 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d3dcff-05ea-4ab2-910b-7a2206274a0a-kube-api-access-5hxq2" pod "51d3dcff-05ea-4ab2-910b-7a2206274a0a" (UID: "51d3dcff-05ea-4ab2-910b-7a2206274a0a"). InnerVolumeSpecName "kube-api-access-5hxq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:45:09.449820 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.449799 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-kube-api-access-vbpgj" pod "5ac41e6f-2f9d-4d2c-af89-223460ffb22f" (UID: "5ac41e6f-2f9d-4d2c-af89-223460ffb22f"). InnerVolumeSpecName "kube-api-access-vbpgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:45:09.454923 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.454894 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-util" pod "51d3dcff-05ea-4ab2-910b-7a2206274a0a" (UID: "51d3dcff-05ea-4ab2-910b-7a2206274a0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:09.455275 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.455253 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-util" pod "5ac41e6f-2f9d-4d2c-af89-223460ffb22f" (UID: "5ac41e6f-2f9d-4d2c-af89-223460ffb22f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:09.547374 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.547338 5015 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-util\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:09.547374 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.547370 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hxq2\" (UniqueName: \"kubernetes.io/projected/51d3dcff-05ea-4ab2-910b-7a2206274a0a-kube-api-access-5hxq2\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:09.547374 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.547380 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbpgj\" (UniqueName: \"kubernetes.io/projected/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-kube-api-access-vbpgj\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:09.547693 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.547396 5015 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:09.547693 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.547409 5015 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ac41e6f-2f9d-4d2c-af89-223460ffb22f-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:09.547693 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:09.547417 5015 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51d3dcff-05ea-4ab2-910b-7a2206274a0a-util\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:10.176889 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.176857 5015 generic.go:356] "Generic (PLEG): container finished" podID="efc99055-280d-4def-833b-b73c4e1a8b8a" containerID="a50c6c730082cd3c3dfd9d465ed853dea6bed5b585c5cd59216509849015d0f3" exitCode=0 Nov 03 08:45:10.177361 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.176904 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" event={"ID":"efc99055-280d-4def-833b-b73c4e1a8b8a","Type":"ContainerDied","Data":"a50c6c730082cd3c3dfd9d465ed853dea6bed5b585c5cd59216509849015d0f3"} Nov 03 08:45:10.179450 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.179425 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" event={"ID":"8f9ac048-a91e-4ba9-8299-4f241d0734eb","Type":"ContainerStarted","Data":"82a40ddd737720613d0c58ed7d2dd2884b5ce5680947dde8dc684c6b5ee0b4a7"} Nov 03 08:45:10.181467 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.181442 5015 generic.go:356] "Generic (PLEG): container finished" podID="73ac1ca1-e946-4c1b-92e5-688169058619" containerID="92e2469eb6ca245362c5c3551b9c021192bb0cc391eb555f73e374535da78b80" exitCode=0 Nov 03 08:45:10.181617 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.181512 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" event={"ID":"73ac1ca1-e946-4c1b-92e5-688169058619","Type":"ContainerDied","Data":"92e2469eb6ca245362c5c3551b9c021192bb0cc391eb555f73e374535da78b80"} Nov 03 08:45:10.183523 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.183500 5015 generic.go:356] "Generic (PLEG): container finished" podID="68e88181-3bd5-4803-b24d-a69734199601" containerID="3140ba5d18f221e7136021eb5e526c53dae9642c4e4efccd184dc355b9cbfb9a" exitCode=0 Nov 03 08:45:10.183634 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.183583 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" event={"ID":"68e88181-3bd5-4803-b24d-a69734199601","Type":"ContainerDied","Data":"3140ba5d18f221e7136021eb5e526c53dae9642c4e4efccd184dc355b9cbfb9a"} Nov 03 08:45:10.185294 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.185273 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" event={"ID":"51d3dcff-05ea-4ab2-910b-7a2206274a0a","Type":"ContainerDied","Data":"e798fedee070f201d778f34953621e6048b97238bd888ce6256eb8bff3a2d2b7"} Nov 03 08:45:10.185399 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.185312 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e798fedee070f201d778f34953621e6048b97238bd888ce6256eb8bff3a2d2b7" Nov 03 08:45:10.185399 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.185282 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q6lcd" Nov 03 08:45:10.187229 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.187211 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" Nov 03 08:45:10.187318 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.187244 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqgf8v" event={"ID":"5ac41e6f-2f9d-4d2c-af89-223460ffb22f","Type":"ContainerDied","Data":"09b4becfa552a880a00699b2505f7f3953bef47959e5d221dce83fa343c3090c"} Nov 03 08:45:10.187318 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.187271 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09b4becfa552a880a00699b2505f7f3953bef47959e5d221dce83fa343c3090c" Nov 03 08:45:10.426127 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.426100 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:45:10.429291 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.429267 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:45:10.432669 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.432647 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:45:10.556644 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.556553 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-kube-api-access-68zfq\" (UniqueName: \"kubernetes.io/projected/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-kube-api-access-68zfq\") pod \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " Nov 03 08:45:10.556644 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.556625 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-kube-api-access-d498h\" (UniqueName: \"kubernetes.io/projected/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-kube-api-access-d498h\") pod \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " Nov 03 08:45:10.556869 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.556660 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-bundle\") pod \"a2b5e45f-e030-446a-a8ca-2decb19a8384\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " Nov 03 08:45:10.556869 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.556689 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-bundle\") pod \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " Nov 03 08:45:10.556869 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.556731 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-util\" (UniqueName: \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-util\") pod \"a2b5e45f-e030-446a-a8ca-2decb19a8384\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " Nov 03 08:45:10.556869 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.556762 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-util\" (UniqueName: \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-util\") pod \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " Nov 03 08:45:10.556869 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.556799 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b5e45f-e030-446a-a8ca-2decb19a8384-kube-api-access-wldqm\" (UniqueName: \"kubernetes.io/projected/a2b5e45f-e030-446a-a8ca-2decb19a8384-kube-api-access-wldqm\") pod \"a2b5e45f-e030-446a-a8ca-2decb19a8384\" (UID: \"a2b5e45f-e030-446a-a8ca-2decb19a8384\") " Nov 03 08:45:10.556869 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.556848 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-bundle\") pod \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\" (UID: \"a92f1ea2-322b-4fde-9525-2a16df2bd6b7\") " Nov 03 08:45:10.557205 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.557179 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-util\" (UniqueName: \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-util\") pod \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\" (UID: \"5b53b3f3-51f9-4b7e-ab25-efe7c4938089\") " Nov 03 08:45:10.557312 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.557280 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-bundle" pod "5b53b3f3-51f9-4b7e-ab25-efe7c4938089" (UID: "5b53b3f3-51f9-4b7e-ab25-efe7c4938089"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:10.557861 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.557836 5015 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:10.558063 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.558029 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-bundle" pod "a2b5e45f-e030-446a-a8ca-2decb19a8384" (UID: "a2b5e45f-e030-446a-a8ca-2decb19a8384"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:10.558305 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.558277 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-bundle" pod "a92f1ea2-322b-4fde-9525-2a16df2bd6b7" (UID: "a92f1ea2-322b-4fde-9525-2a16df2bd6b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:10.559247 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.559223 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-kube-api-access-d498h" pod "a92f1ea2-322b-4fde-9525-2a16df2bd6b7" (UID: "a92f1ea2-322b-4fde-9525-2a16df2bd6b7"). InnerVolumeSpecName "kube-api-access-d498h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:45:10.559345 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.559324 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-kube-api-access-68zfq" pod "5b53b3f3-51f9-4b7e-ab25-efe7c4938089" (UID: "5b53b3f3-51f9-4b7e-ab25-efe7c4938089"). InnerVolumeSpecName "kube-api-access-68zfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:45:10.559783 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.559759 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b5e45f-e030-446a-a8ca-2decb19a8384-kube-api-access-wldqm" pod "a2b5e45f-e030-446a-a8ca-2decb19a8384" (UID: "a2b5e45f-e030-446a-a8ca-2decb19a8384"). InnerVolumeSpecName "kube-api-access-wldqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:45:10.562606 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.562587 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-util" pod "5b53b3f3-51f9-4b7e-ab25-efe7c4938089" (UID: "5b53b3f3-51f9-4b7e-ab25-efe7c4938089"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:10.563640 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.563620 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-util" pod "a92f1ea2-322b-4fde-9525-2a16df2bd6b7" (UID: "a92f1ea2-322b-4fde-9525-2a16df2bd6b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:10.563791 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.563773 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-util" pod "a2b5e45f-e030-446a-a8ca-2decb19a8384" (UID: "a2b5e45f-e030-446a-a8ca-2decb19a8384"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:10.659057 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.659024 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d498h\" (UniqueName: \"kubernetes.io/projected/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-kube-api-access-d498h\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:10.659057 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.659052 5015 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:10.659057 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.659061 5015 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b5e45f-e030-446a-a8ca-2decb19a8384-util\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:10.659279 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.659069 5015 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-util\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:10.659279 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.659078 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wldqm\" (UniqueName: \"kubernetes.io/projected/a2b5e45f-e030-446a-a8ca-2decb19a8384-kube-api-access-wldqm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:10.659279 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.659086 5015 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a92f1ea2-322b-4fde-9525-2a16df2bd6b7-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:10.659279 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.659094 5015 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-util\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:10.659279 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.659101 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68zfq\" (UniqueName: \"kubernetes.io/projected/5b53b3f3-51f9-4b7e-ab25-efe7c4938089-kube-api-access-68zfq\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:10.824686 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:10.824584 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" podStartSLOduration=4.925855179 podStartE2EDuration="41.824567748s" totalImagesPullingTime="36.898712569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:10.759027543 +0000 UTC m=+343.683969782" watchObservedRunningTime="2026-11-03 08:45:10.824567748 +0000 UTC m=+343.749509965" Nov 03 08:45:11.194124 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.194032 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" event={"ID":"a2b5e45f-e030-446a-a8ca-2decb19a8384","Type":"ContainerDied","Data":"ec94d70e4e72e8760d0d2a19e946aa42ad78a0467a8b7497c9c2b665ef404d35"} Nov 03 08:45:11.194124 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.194116 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec94d70e4e72e8760d0d2a19e946aa42ad78a0467a8b7497c9c2b665ef404d35" Nov 03 08:45:11.194124 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.194051 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agwczx" Nov 03 08:45:11.195991 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.195969 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" Nov 03 08:45:11.195991 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.195985 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czz4vb" event={"ID":"a92f1ea2-322b-4fde-9525-2a16df2bd6b7","Type":"ContainerDied","Data":"1e62a218331c31da8058faf7864f100d0726d087ac744e7809df0f75bc8c8ad0"} Nov 03 08:45:11.196159 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.196023 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e62a218331c31da8058faf7864f100d0726d087ac744e7809df0f75bc8c8ad0" Nov 03 08:45:11.198331 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.198308 5015 generic.go:356] "Generic (PLEG): container finished" podID="8f9ac048-a91e-4ba9-8299-4f241d0734eb" containerID="82a40ddd737720613d0c58ed7d2dd2884b5ce5680947dde8dc684c6b5ee0b4a7" exitCode=0 Nov 03 08:45:11.200716 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.200698 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" Nov 03 08:45:11.204890 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.204861 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" event={"ID":"8f9ac048-a91e-4ba9-8299-4f241d0734eb","Type":"ContainerDied","Data":"82a40ddd737720613d0c58ed7d2dd2884b5ce5680947dde8dc684c6b5ee0b4a7"} Nov 03 08:45:11.204993 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.204912 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75r9rh" event={"ID":"5b53b3f3-51f9-4b7e-ab25-efe7c4938089","Type":"ContainerDied","Data":"9de3ad68e3c4a45c8f821bfafef3fc88fce1190e80932bc5b76372ea03ecdec1"} Nov 03 08:45:11.204993 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.204936 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9de3ad68e3c4a45c8f821bfafef3fc88fce1190e80932bc5b76372ea03ecdec1" Nov 03 08:45:11.340301 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.340278 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:45:11.384887 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.384863 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:45:11.387717 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.387696 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:45:11.470272 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.470185 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc99055-280d-4def-833b-b73c4e1a8b8a-kube-api-access-bnnrl\" (UniqueName: \"kubernetes.io/projected/efc99055-280d-4def-833b-b73c4e1a8b8a-kube-api-access-bnnrl\") pod \"efc99055-280d-4def-833b-b73c4e1a8b8a\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " Nov 03 08:45:11.470272 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.470260 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ac1ca1-e946-4c1b-92e5-688169058619-kube-api-access-bkkd6\" (UniqueName: \"kubernetes.io/projected/73ac1ca1-e946-4c1b-92e5-688169058619-kube-api-access-bkkd6\") pod \"73ac1ca1-e946-4c1b-92e5-688169058619\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " Nov 03 08:45:11.470551 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.470285 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-util\" (UniqueName: \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-util\") pod \"efc99055-280d-4def-833b-b73c4e1a8b8a\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " Nov 03 08:45:11.470551 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.470319 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-bundle\") pod \"efc99055-280d-4def-833b-b73c4e1a8b8a\" (UID: \"efc99055-280d-4def-833b-b73c4e1a8b8a\") " Nov 03 08:45:11.470551 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.470349 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-bundle\" (UniqueName: \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-bundle\") pod \"73ac1ca1-e946-4c1b-92e5-688169058619\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " Nov 03 08:45:11.470551 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.470390 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-util\" (UniqueName: \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-util\") pod \"73ac1ca1-e946-4c1b-92e5-688169058619\" (UID: \"73ac1ca1-e946-4c1b-92e5-688169058619\") " Nov 03 08:45:11.470551 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.470420 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-bundle\" (UniqueName: \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-bundle\") pod \"68e88181-3bd5-4803-b24d-a69734199601\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " Nov 03 08:45:11.470551 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.470454 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-util\" (UniqueName: \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-util\") pod \"68e88181-3bd5-4803-b24d-a69734199601\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " Nov 03 08:45:11.470551 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.470509 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e88181-3bd5-4803-b24d-a69734199601-kube-api-access-zh24t\" (UniqueName: \"kubernetes.io/projected/68e88181-3bd5-4803-b24d-a69734199601-kube-api-access-zh24t\") pod \"68e88181-3bd5-4803-b24d-a69734199601\" (UID: \"68e88181-3bd5-4803-b24d-a69734199601\") " Nov 03 08:45:11.471165 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.471140 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-bundle" pod "efc99055-280d-4def-833b-b73c4e1a8b8a" (UID: "efc99055-280d-4def-833b-b73c4e1a8b8a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:11.471258 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.471221 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-bundle" pod "73ac1ca1-e946-4c1b-92e5-688169058619" (UID: "73ac1ca1-e946-4c1b-92e5-688169058619"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:11.471650 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.471621 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-bundle" pod "68e88181-3bd5-4803-b24d-a69734199601" (UID: "68e88181-3bd5-4803-b24d-a69734199601"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:11.472813 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.472789 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ac1ca1-e946-4c1b-92e5-688169058619-kube-api-access-bkkd6" pod "73ac1ca1-e946-4c1b-92e5-688169058619" (UID: "73ac1ca1-e946-4c1b-92e5-688169058619"). InnerVolumeSpecName "kube-api-access-bkkd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:45:11.473233 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.473205 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc99055-280d-4def-833b-b73c4e1a8b8a-kube-api-access-bnnrl" pod "efc99055-280d-4def-833b-b73c4e1a8b8a" (UID: "efc99055-280d-4def-833b-b73c4e1a8b8a"). InnerVolumeSpecName "kube-api-access-bnnrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:45:11.473388 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.473370 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e88181-3bd5-4803-b24d-a69734199601-kube-api-access-zh24t" pod "68e88181-3bd5-4803-b24d-a69734199601" (UID: "68e88181-3bd5-4803-b24d-a69734199601"). InnerVolumeSpecName "kube-api-access-zh24t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:45:11.476585 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.476565 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-util" pod "68e88181-3bd5-4803-b24d-a69734199601" (UID: "68e88181-3bd5-4803-b24d-a69734199601"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:11.476710 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.476691 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-util" pod "efc99055-280d-4def-833b-b73c4e1a8b8a" (UID: "efc99055-280d-4def-833b-b73c4e1a8b8a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:11.478417 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.478397 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-util" pod "73ac1ca1-e946-4c1b-92e5-688169058619" (UID: "73ac1ca1-e946-4c1b-92e5-688169058619"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:11.571591 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.571552 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnnrl\" (UniqueName: \"kubernetes.io/projected/efc99055-280d-4def-833b-b73c4e1a8b8a-kube-api-access-bnnrl\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:11.571591 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.571578 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkkd6\" (UniqueName: \"kubernetes.io/projected/73ac1ca1-e946-4c1b-92e5-688169058619-kube-api-access-bkkd6\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:11.571591 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.571588 5015 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-util\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:11.571591 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.571597 5015 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efc99055-280d-4def-833b-b73c4e1a8b8a-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:11.571866 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.571605 5015 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:11.571866 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.571612 5015 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73ac1ca1-e946-4c1b-92e5-688169058619-util\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:11.571866 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.571620 5015 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:11.571866 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.571626 5015 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68e88181-3bd5-4803-b24d-a69734199601-util\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:11.571866 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:11.571635 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh24t\" (UniqueName: \"kubernetes.io/projected/68e88181-3bd5-4803-b24d-a69734199601-kube-api-access-zh24t\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:12.207921 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.207887 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" Nov 03 08:45:12.207921 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.207897 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvqjf" event={"ID":"efc99055-280d-4def-833b-b73c4e1a8b8a","Type":"ContainerDied","Data":"8096c19526b74154d402b5df1fc6ada78c316995c2a4f8f4c98caf6e481a4f2a"} Nov 03 08:45:12.208358 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.207943 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8096c19526b74154d402b5df1fc6ada78c316995c2a4f8f4c98caf6e481a4f2a" Nov 03 08:45:12.210178 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.210159 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" Nov 03 08:45:12.210178 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.210163 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66zlkh" event={"ID":"73ac1ca1-e946-4c1b-92e5-688169058619","Type":"ContainerDied","Data":"8afe69e396ecc56f38834892b30af60499d0719f7bf2abcc8ab2cf4349b450df"} Nov 03 08:45:12.210325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.210202 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8afe69e396ecc56f38834892b30af60499d0719f7bf2abcc8ab2cf4349b450df" Nov 03 08:45:12.212532 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.212511 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" Nov 03 08:45:12.212532 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.212517 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125dwc" event={"ID":"68e88181-3bd5-4803-b24d-a69734199601","Type":"ContainerDied","Data":"3557cc8b8b634a0f9b920bf769ef82dd0e7ad4110796ebae8070a8d4930b2ea8"} Nov 03 08:45:12.212676 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.212554 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3557cc8b8b634a0f9b920bf769ef82dd0e7ad4110796ebae8070a8d4930b2ea8" Nov 03 08:45:12.343406 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.343378 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:45:12.477890 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.477862 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9ac048-a91e-4ba9-8299-4f241d0734eb-kube-api-access-9jvr6\" (UniqueName: \"kubernetes.io/projected/8f9ac048-a91e-4ba9-8299-4f241d0734eb-kube-api-access-9jvr6\") pod \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " Nov 03 08:45:12.478061 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.477917 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-util\" (UniqueName: \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-util\") pod \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " Nov 03 08:45:12.478061 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.477955 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-bundle\") pod \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\" (UID: \"8f9ac048-a91e-4ba9-8299-4f241d0734eb\") " Nov 03 08:45:12.478720 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.478696 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-bundle" pod "8f9ac048-a91e-4ba9-8299-4f241d0734eb" (UID: "8f9ac048-a91e-4ba9-8299-4f241d0734eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:12.480277 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.480260 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9ac048-a91e-4ba9-8299-4f241d0734eb-kube-api-access-9jvr6" pod "8f9ac048-a91e-4ba9-8299-4f241d0734eb" (UID: "8f9ac048-a91e-4ba9-8299-4f241d0734eb"). InnerVolumeSpecName "kube-api-access-9jvr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:45:12.484138 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.484118 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-util" pod "8f9ac048-a91e-4ba9-8299-4f241d0734eb" (UID: "8f9ac048-a91e-4ba9-8299-4f241d0734eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:45:12.578672 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.578636 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvr6\" (UniqueName: \"kubernetes.io/projected/8f9ac048-a91e-4ba9-8299-4f241d0734eb-kube-api-access-9jvr6\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:12.578672 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.578670 5015 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-util\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:12.578672 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:12.578680 5015 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f9ac048-a91e-4ba9-8299-4f241d0734eb-bundle\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:45:13.219128 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:13.219092 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" event={"ID":"8f9ac048-a91e-4ba9-8299-4f241d0734eb","Type":"ContainerDied","Data":"81e9c97b30401a070f21be1792bd3f88862693f7071eaf75d460277557861be4"} Nov 03 08:45:13.219547 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:13.219169 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81e9c97b30401a070f21be1792bd3f88862693f7071eaf75d460277557861be4" Nov 03 08:45:13.219547 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:13.219106 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvv6rh" Nov 03 08:45:16.901829 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:45:16.901795 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:45:24.949834 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.949793 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt"] Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950290 5015 cpu_manager.go:414] "containerMap: removing container" podUID="51d3dcff-05ea-4ab2-910b-7a2206274a0a" containerName="extract" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950309 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d3dcff-05ea-4ab2-910b-7a2206274a0a" containerName="extract" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950331 5015 cpu_manager.go:414] "containerMap: removing container" podUID="efc99055-280d-4def-833b-b73c4e1a8b8a" containerName="util" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950338 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc99055-280d-4def-833b-b73c4e1a8b8a" containerName="util" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950351 5015 cpu_manager.go:414] "containerMap: removing container" podUID="e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" containerName="util" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950359 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" containerName="util" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950371 5015 cpu_manager.go:414] "containerMap: removing container" podUID="efc99055-280d-4def-833b-b73c4e1a8b8a" containerName="pull" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950377 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc99055-280d-4def-833b-b73c4e1a8b8a" containerName="pull" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950392 5015 cpu_manager.go:414] "containerMap: removing container" podUID="68e88181-3bd5-4803-b24d-a69734199601" containerName="util" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950399 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e88181-3bd5-4803-b24d-a69734199601" containerName="util" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950410 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a2b5e45f-e030-446a-a8ca-2decb19a8384" containerName="util" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950417 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b5e45f-e030-446a-a8ca-2decb19a8384" containerName="util" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950426 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5b53b3f3-51f9-4b7e-ab25-efe7c4938089" containerName="util" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950432 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b53b3f3-51f9-4b7e-ab25-efe7c4938089" containerName="util" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950441 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a2b5e45f-e030-446a-a8ca-2decb19a8384" containerName="extract" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950447 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b5e45f-e030-446a-a8ca-2decb19a8384" containerName="extract" Nov 03 08:45:24.950453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950457 5015 cpu_manager.go:414] "containerMap: removing container" podUID="e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950463 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950507 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5ac41e6f-2f9d-4d2c-af89-223460ffb22f" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950514 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac41e6f-2f9d-4d2c-af89-223460ffb22f" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950523 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a92f1ea2-322b-4fde-9525-2a16df2bd6b7" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950530 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92f1ea2-322b-4fde-9525-2a16df2bd6b7" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950543 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8f9ac048-a91e-4ba9-8299-4f241d0734eb" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950549 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9ac048-a91e-4ba9-8299-4f241d0734eb" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950560 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8f9ac048-a91e-4ba9-8299-4f241d0734eb" containerName="util" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950567 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9ac048-a91e-4ba9-8299-4f241d0734eb" containerName="util" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950576 5015 cpu_manager.go:414] "containerMap: removing container" podUID="51d3dcff-05ea-4ab2-910b-7a2206274a0a" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950583 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d3dcff-05ea-4ab2-910b-7a2206274a0a" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950593 5015 cpu_manager.go:414] "containerMap: removing container" podUID="efc99055-280d-4def-833b-b73c4e1a8b8a" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950600 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc99055-280d-4def-833b-b73c4e1a8b8a" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950611 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a92f1ea2-322b-4fde-9525-2a16df2bd6b7" containerName="util" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950618 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92f1ea2-322b-4fde-9525-2a16df2bd6b7" containerName="util" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950629 5015 cpu_manager.go:414] "containerMap: removing container" podUID="73ac1ca1-e946-4c1b-92e5-688169058619" containerName="util" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950636 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ac1ca1-e946-4c1b-92e5-688169058619" containerName="util" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950655 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5ac41e6f-2f9d-4d2c-af89-223460ffb22f" containerName="util" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950664 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac41e6f-2f9d-4d2c-af89-223460ffb22f" containerName="util" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950673 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a2b5e45f-e030-446a-a8ca-2decb19a8384" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950680 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b5e45f-e030-446a-a8ca-2decb19a8384" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950690 5015 cpu_manager.go:414] "containerMap: removing container" podUID="e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950697 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950710 5015 cpu_manager.go:414] "containerMap: removing container" podUID="51d3dcff-05ea-4ab2-910b-7a2206274a0a" containerName="util" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950717 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d3dcff-05ea-4ab2-910b-7a2206274a0a" containerName="util" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950727 5015 cpu_manager.go:414] "containerMap: removing container" podUID="73ac1ca1-e946-4c1b-92e5-688169058619" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950733 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ac1ca1-e946-4c1b-92e5-688169058619" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950743 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5ac41e6f-2f9d-4d2c-af89-223460ffb22f" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950750 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac41e6f-2f9d-4d2c-af89-223460ffb22f" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950758 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5b53b3f3-51f9-4b7e-ab25-efe7c4938089" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950765 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b53b3f3-51f9-4b7e-ab25-efe7c4938089" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950774 5015 cpu_manager.go:414] "containerMap: removing container" podUID="73ac1ca1-e946-4c1b-92e5-688169058619" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950780 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ac1ca1-e946-4c1b-92e5-688169058619" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950791 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8f9ac048-a91e-4ba9-8299-4f241d0734eb" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950798 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9ac048-a91e-4ba9-8299-4f241d0734eb" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950813 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5b53b3f3-51f9-4b7e-ab25-efe7c4938089" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950819 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b53b3f3-51f9-4b7e-ab25-efe7c4938089" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950828 5015 cpu_manager.go:414] "containerMap: removing container" podUID="68e88181-3bd5-4803-b24d-a69734199601" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950835 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e88181-3bd5-4803-b24d-a69734199601" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950845 5015 cpu_manager.go:414] "containerMap: removing container" podUID="68e88181-3bd5-4803-b24d-a69734199601" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950851 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e88181-3bd5-4803-b24d-a69734199601" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950864 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a92f1ea2-322b-4fde-9525-2a16df2bd6b7" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950870 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92f1ea2-322b-4fde-9525-2a16df2bd6b7" containerName="pull" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950961 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9af3f54-4abd-49f5-a83b-cd3c80ab19b3" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950972 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac41e6f-2f9d-4d2c-af89-223460ffb22f" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950983 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b53b3f3-51f9-4b7e-ab25-efe7c4938089" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950990 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ac1ca1-e946-4c1b-92e5-688169058619" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.950998 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f9ac048-a91e-4ba9-8299-4f241d0734eb" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.951006 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e88181-3bd5-4803-b24d-a69734199601" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.951016 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d3dcff-05ea-4ab2-910b-7a2206274a0a" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.951024 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92f1ea2-322b-4fde-9525-2a16df2bd6b7" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.951035 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b5e45f-e030-446a-a8ca-2decb19a8384" containerName="extract" Nov 03 08:45:24.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.951046 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc99055-280d-4def-833b-b73c4e1a8b8a" containerName="extract" Nov 03 08:45:24.954589 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:24.954569 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.063270 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.063238 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt"] Nov 03 08:45:25.087234 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.087201 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 08:45:25.087492 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.087451 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.087543 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.087509 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-metrics-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.087599 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.087586 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-tmp\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.087637 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.087621 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.087676 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.087646 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrqc\" (UniqueName: \"kubernetes.io/projected/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-kube-api-access-cbrqc\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.087676 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.087663 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-manifests\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.087860 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.087836 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 08:45:25.088115 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.088097 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 08:45:25.088232 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.088212 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 08:45:25.088297 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.088268 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 08:45:25.088336 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.088297 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-tbsxf\"" Nov 03 08:45:25.188822 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.188777 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrqc\" (UniqueName: \"kubernetes.io/projected/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-kube-api-access-cbrqc\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.188822 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.188814 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-manifests\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.189098 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.188860 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.189098 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.188884 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-metrics-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.189098 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.188903 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-tmp\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.189098 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.188932 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.189444 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.189413 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-manifests\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.191488 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.191452 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-tmp\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.191768 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.191744 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.191930 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.191914 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-metrics-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.191990 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.191959 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.252577 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.252537 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrqc\" (UniqueName: \"kubernetes.io/projected/0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2-kube-api-access-cbrqc\") pod \"opendatahub-operator-controller-manager-77df88dcb4-dplnt\" (UID: \"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.268390 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.268357 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:25.763664 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:25.763638 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt"] Nov 03 08:45:25.765858 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:45:25.765831 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e602fbb_7c1e_41ef_8c9e_a174df7dc1d2.slice/crio-de5314e2abc56a0a147a375f2540c0d23f44e750564cbacc0543dbc709577fc1 WatchSource:0}: Error finding container de5314e2abc56a0a147a375f2540c0d23f44e750564cbacc0543dbc709577fc1: Status 404 returned error can't find the container with id de5314e2abc56a0a147a375f2540c0d23f44e750564cbacc0543dbc709577fc1 Nov 03 08:45:26.282868 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:26.282834 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" event={"ID":"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2","Type":"ContainerStarted","Data":"de5314e2abc56a0a147a375f2540c0d23f44e750564cbacc0543dbc709577fc1"} Nov 03 08:45:29.535698 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:29.535664 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rc42m"] Nov 03 08:45:29.541393 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:29.541366 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rc42m" Nov 03 08:45:29.596224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:29.596191 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 08:45:29.645862 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:29.645830 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kmvkl\"" Nov 03 08:45:29.646056 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:29.645869 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 08:45:29.727410 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:29.727364 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnp9r\" (UniqueName: \"kubernetes.io/projected/1786e9e7-2202-4fce-aaf4-2f0b30932688-kube-api-access-pnp9r\") pod \"authorino-operator-7b7bb4bc77-rc42m\" (UID: \"1786e9e7-2202-4fce-aaf4-2f0b30932688\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rc42m" Nov 03 08:45:29.757014 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:29.756976 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rc42m"] Nov 03 08:45:29.828241 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:29.828157 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnp9r\" (UniqueName: \"kubernetes.io/projected/1786e9e7-2202-4fce-aaf4-2f0b30932688-kube-api-access-pnp9r\") pod \"authorino-operator-7b7bb4bc77-rc42m\" (UID: \"1786e9e7-2202-4fce-aaf4-2f0b30932688\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rc42m" Nov 03 08:45:29.895914 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:29.895877 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnp9r\" (UniqueName: \"kubernetes.io/projected/1786e9e7-2202-4fce-aaf4-2f0b30932688-kube-api-access-pnp9r\") pod \"authorino-operator-7b7bb4bc77-rc42m\" (UID: \"1786e9e7-2202-4fce-aaf4-2f0b30932688\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rc42m" Nov 03 08:45:30.154782 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:30.154487 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rc42m" Nov 03 08:45:30.556774 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:30.556735 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rc42m"] Nov 03 08:45:30.591016 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:45:30.590981 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1786e9e7_2202_4fce_aaf4_2f0b30932688.slice/crio-6ad90aaab3d5a948909162abf438931606f3e0d6bf6bcd3337c672f4970e5490 WatchSource:0}: Error finding container 6ad90aaab3d5a948909162abf438931606f3e0d6bf6bcd3337c672f4970e5490: Status 404 returned error can't find the container with id 6ad90aaab3d5a948909162abf438931606f3e0d6bf6bcd3337c672f4970e5490 Nov 03 08:45:31.315209 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:31.315181 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" event={"ID":"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2","Type":"ContainerStarted","Data":"d0bc18ef44f74ede9bf7c6fd8ec653eec9b34d87742b7892163ba316fb587dfd"} Nov 03 08:45:31.316525 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:31.316501 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rc42m" event={"ID":"1786e9e7-2202-4fce-aaf4-2f0b30932688","Type":"ContainerStarted","Data":"6ad90aaab3d5a948909162abf438931606f3e0d6bf6bcd3337c672f4970e5490"} Nov 03 08:45:32.324059 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:32.324025 5015 generic.go:356] "Generic (PLEG): container finished" podID="0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2" containerID="d0bc18ef44f74ede9bf7c6fd8ec653eec9b34d87742b7892163ba316fb587dfd" exitCode=0 Nov 03 08:45:32.324500 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:32.324110 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" event={"ID":"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2","Type":"ContainerDied","Data":"d0bc18ef44f74ede9bf7c6fd8ec653eec9b34d87742b7892163ba316fb587dfd"} Nov 03 08:45:33.330516 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:33.330452 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" event={"ID":"0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2","Type":"ContainerStarted","Data":"7dbbe494b9cf179cf6039ebf20b259603a2225bb23a6691fd6014f466eca250e"} Nov 03 08:45:33.331003 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:33.330611 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:33.625929 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:33.625850 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" podStartSLOduration=2.878683321 podStartE2EDuration="9.625834999s" totalImagesPullingTime="6.244627556s" totalInitContainerRuntime="502.524122ms" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:33.575182728 +0000 UTC m=+366.500124964" watchObservedRunningTime="2026-11-03 08:45:33.625834999 +0000 UTC m=+366.550777277" Nov 03 08:45:37.013190 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.013155 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf"] Nov 03 08:45:37.018867 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.018846 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf" Nov 03 08:45:37.066105 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.066080 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf"] Nov 03 08:45:37.085242 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.085210 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf\" (UID: \"f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf" Nov 03 08:45:37.085397 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.085278 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgczc\" (UniqueName: \"kubernetes.io/projected/f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e-kube-api-access-wgczc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf\" (UID: \"f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf" Nov 03 08:45:37.114019 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.113993 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:45:37.114019 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.113999 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 08:45:37.114155 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.114074 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mj85x\"" Nov 03 08:45:37.185976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.185944 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgczc\" (UniqueName: \"kubernetes.io/projected/f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e-kube-api-access-wgczc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf\" (UID: \"f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf" Nov 03 08:45:37.186119 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.186000 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf\" (UID: \"f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf" Nov 03 08:45:37.186344 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.186328 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf\" (UID: \"f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf" Nov 03 08:45:37.249928 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.249898 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgczc\" (UniqueName: \"kubernetes.io/projected/f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e-kube-api-access-wgczc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf\" (UID: \"f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf" Nov 03 08:45:37.330528 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.330456 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf" Nov 03 08:45:37.836947 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.836921 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf"] Nov 03 08:45:37.838446 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:45:37.838415 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e4e2b1_9f19_4f46_ae43_d7e5c71db30e.slice/crio-498fe05a370b3607f2b80562820b7d58cff1b96588ed97ec722ab07cb33b7a04 WatchSource:0}: Error finding container 498fe05a370b3607f2b80562820b7d58cff1b96588ed97ec722ab07cb33b7a04: Status 404 returned error can't find the container with id 498fe05a370b3607f2b80562820b7d58cff1b96588ed97ec722ab07cb33b7a04 Nov 03 08:45:37.887647 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.887614 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8"] Nov 03 08:45:37.890350 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.890323 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 08:45:37.890657 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.890639 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5j6h\" (UniqueName: \"kubernetes.io/projected/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-kube-api-access-j5j6h\") pod \"kuadrant-operator-controller-manager-678d87766f-8bqv8\" (UID: \"05e4fc7a-c29d-4829-84f6-06cd99b8dae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 08:45:37.890768 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.890751 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8bqv8\" (UID: \"05e4fc7a-c29d-4829-84f6-06cd99b8dae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 08:45:37.921097 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.921067 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8"] Nov 03 08:45:37.947272 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.947240 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-s4rff\"" Nov 03 08:45:37.991817 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.991790 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8bqv8\" (UID: \"05e4fc7a-c29d-4829-84f6-06cd99b8dae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 08:45:37.991963 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.991855 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5j6h\" (UniqueName: \"kubernetes.io/projected/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-kube-api-access-j5j6h\") pod \"kuadrant-operator-controller-manager-678d87766f-8bqv8\" (UID: \"05e4fc7a-c29d-4829-84f6-06cd99b8dae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 08:45:37.992182 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:37.992161 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8bqv8\" (UID: \"05e4fc7a-c29d-4829-84f6-06cd99b8dae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 08:45:38.054447 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:38.054419 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5j6h\" (UniqueName: \"kubernetes.io/projected/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-kube-api-access-j5j6h\") pod \"kuadrant-operator-controller-manager-678d87766f-8bqv8\" (UID: \"05e4fc7a-c29d-4829-84f6-06cd99b8dae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 08:45:38.202163 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:38.202085 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 08:45:38.354739 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:38.354706 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf" event={"ID":"f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e","Type":"ContainerStarted","Data":"498fe05a370b3607f2b80562820b7d58cff1b96588ed97ec722ab07cb33b7a04"} Nov 03 08:45:38.625956 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:38.625923 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8"] Nov 03 08:45:38.630019 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:45:38.629992 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e4fc7a_c29d_4829_84f6_06cd99b8dae1.slice/crio-21107d4b528c620bd6959fcb096b29bd52496e23b5218d250c25d4931a914a69 WatchSource:0}: Error finding container 21107d4b528c620bd6959fcb096b29bd52496e23b5218d250c25d4931a914a69: Status 404 returned error can't find the container with id 21107d4b528c620bd6959fcb096b29bd52496e23b5218d250c25d4931a914a69 Nov 03 08:45:39.361435 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:39.361389 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" event={"ID":"05e4fc7a-c29d-4829-84f6-06cd99b8dae1","Type":"ContainerStarted","Data":"21107d4b528c620bd6959fcb096b29bd52496e23b5218d250c25d4931a914a69"} Nov 03 08:45:41.377867 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:41.377827 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf" event={"ID":"f3e4e2b1-9f19-4f46-ae43-d7e5c71db30e","Type":"ContainerStarted","Data":"db2b28e77f67fdef6977d5566e1c21904586484fce91e3cda9116bbac4667419"} Nov 03 08:45:41.601360 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:41.601302 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2rdsf" podStartSLOduration=2.973449756 podStartE2EDuration="5.601288275s" totalImagesPullingTime="2.627838519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:41.600292374 +0000 UTC m=+374.525234603" watchObservedRunningTime="2026-11-03 08:45:41.601288275 +0000 UTC m=+374.526230489" Nov 03 08:45:43.391571 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:43.391523 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rc42m" event={"ID":"1786e9e7-2202-4fce-aaf4-2f0b30932688","Type":"ContainerStarted","Data":"8557719d424054c5c741004ec7e1fc3221d9bad23b98fae1aa02a72b8641e008"} Nov 03 08:45:43.392026 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:43.391686 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rc42m" Nov 03 08:45:43.513575 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:43.513502 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rc42m" podStartSLOduration=2.264699921 podStartE2EDuration="14.513463498s" totalImagesPullingTime="12.248763577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:43.511518151 +0000 UTC m=+376.436460393" watchObservedRunningTime="2026-11-03 08:45:43.513463498 +0000 UTC m=+376.438405716" Nov 03 08:45:44.337652 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:44.337622 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-dplnt" Nov 03 08:45:46.902024 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:45:46.901985 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:45:47.412966 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:47.412928 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" event={"ID":"05e4fc7a-c29d-4829-84f6-06cd99b8dae1","Type":"ContainerStarted","Data":"d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9"} Nov 03 08:45:47.413175 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:47.413130 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 08:45:47.634093 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:47.634038 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" podStartSLOduration=2.905092099 podStartE2EDuration="10.634022271s" totalImagesPullingTime="7.728930172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:47.581612883 +0000 UTC m=+380.506555126" watchObservedRunningTime="2026-11-03 08:45:47.634022271 +0000 UTC m=+380.558964488" Nov 03 08:45:54.399643 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:54.399605 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rc42m" Nov 03 08:45:56.289148 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.289107 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zwxdh"] Nov 03 08:45:56.292595 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.292576 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:45:56.349899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.349860 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zwxdh"] Nov 03 08:45:56.354691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.354661 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a3dd0096-28c3-4017-9056-54b721df8abe-config-file\") pod \"limitador-limitador-74dfcfb894-zwxdh\" (UID: \"a3dd0096-28c3-4017-9056-54b721df8abe\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:45:56.354837 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.354718 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h5fg\" (UniqueName: \"kubernetes.io/projected/a3dd0096-28c3-4017-9056-54b721df8abe-kube-api-access-5h5fg\") pod \"limitador-limitador-74dfcfb894-zwxdh\" (UID: \"a3dd0096-28c3-4017-9056-54b721df8abe\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:45:56.397968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.397934 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-znzl4\"" Nov 03 08:45:56.397968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.397934 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 08:45:56.455977 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.455946 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h5fg\" (UniqueName: \"kubernetes.io/projected/a3dd0096-28c3-4017-9056-54b721df8abe-kube-api-access-5h5fg\") pod \"limitador-limitador-74dfcfb894-zwxdh\" (UID: \"a3dd0096-28c3-4017-9056-54b721df8abe\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:45:56.456174 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.456017 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a3dd0096-28c3-4017-9056-54b721df8abe-config-file\") pod \"limitador-limitador-74dfcfb894-zwxdh\" (UID: \"a3dd0096-28c3-4017-9056-54b721df8abe\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:45:56.456725 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.456703 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a3dd0096-28c3-4017-9056-54b721df8abe-config-file\") pod \"limitador-limitador-74dfcfb894-zwxdh\" (UID: \"a3dd0096-28c3-4017-9056-54b721df8abe\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:45:56.515778 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.515745 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h5fg\" (UniqueName: \"kubernetes.io/projected/a3dd0096-28c3-4017-9056-54b721df8abe-kube-api-access-5h5fg\") pod \"limitador-limitador-74dfcfb894-zwxdh\" (UID: \"a3dd0096-28c3-4017-9056-54b721df8abe\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:45:56.538966 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.538931 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zwxdh"] Nov 03 08:45:56.605640 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.605556 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:45:56.988523 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:56.988497 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zwxdh"] Nov 03 08:45:56.990106 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:45:56.990076 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3dd0096_28c3_4017_9056_54b721df8abe.slice/crio-5a5f569a47d2be0761b8d127806143f5bfe1aa06680f19443c4406f745e07da5 WatchSource:0}: Error finding container 5a5f569a47d2be0761b8d127806143f5bfe1aa06680f19443c4406f745e07da5: Status 404 returned error can't find the container with id 5a5f569a47d2be0761b8d127806143f5bfe1aa06680f19443c4406f745e07da5 Nov 03 08:45:57.396027 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.395940 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh"] Nov 03 08:45:57.401153 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.401119 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.459922 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.459879 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qsqtl\"" Nov 03 08:45:57.462346 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.462313 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh"] Nov 03 08:45:57.466804 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.466772 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.466949 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.466816 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.466949 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.466841 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.467054 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.466952 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.467054 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.466984 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.467164 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.467084 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af4db30f-c036-4273-b29e-733cd2dbe65f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.467164 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.467124 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.467268 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.467194 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ndw\" (UniqueName: \"kubernetes.io/projected/af4db30f-c036-4273-b29e-733cd2dbe65f-kube-api-access-d4ndw\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.467268 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.467258 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.469843 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.469811 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" event={"ID":"a3dd0096-28c3-4017-9056-54b721df8abe","Type":"ContainerStarted","Data":"5a5f569a47d2be0761b8d127806143f5bfe1aa06680f19443c4406f745e07da5"} Nov 03 08:45:57.568038 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568003 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ndw\" (UniqueName: \"kubernetes.io/projected/af4db30f-c036-4273-b29e-733cd2dbe65f-kube-api-access-d4ndw\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568038 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568045 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568268 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568073 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568268 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568156 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568268 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568206 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568268 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568263 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568440 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568298 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568440 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568357 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af4db30f-c036-4273-b29e-733cd2dbe65f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568440 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568381 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568627 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568512 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568759 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568732 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568921 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568768 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.568921 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.568885 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.569163 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.569140 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af4db30f-c036-4273-b29e-733cd2dbe65f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.570733 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.570713 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.571098 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.571076 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.628412 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.628373 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af4db30f-c036-4273-b29e-733cd2dbe65f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.628612 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.628590 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ndw\" (UniqueName: \"kubernetes.io/projected/af4db30f-c036-4273-b29e-733cd2dbe65f-kube-api-access-d4ndw\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-86rlh\" (UID: \"af4db30f-c036-4273-b29e-733cd2dbe65f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.723683 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.723650 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:57.980147 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.980068 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-v8d5x"] Nov 03 08:45:57.984851 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:57.984825 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-v8d5x" Nov 03 08:45:58.042502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.042456 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-v8d5x"] Nov 03 08:45:58.069765 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.069716 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-h722q\"" Nov 03 08:45:58.073351 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.073322 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9tjz\" (UniqueName: \"kubernetes.io/projected/8fffcee0-fbee-424d-b719-31f2c91cd1d3-kube-api-access-m9tjz\") pod \"authorino-57f7f8659f-v8d5x\" (UID: \"8fffcee0-fbee-424d-b719-31f2c91cd1d3\") " pod="kuadrant-system/authorino-57f7f8659f-v8d5x" Nov 03 08:45:58.170100 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.170072 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh"] Nov 03 08:45:58.171253 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:45:58.171222 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4db30f_c036_4273_b29e_733cd2dbe65f.slice/crio-16dd09827f583dc0c3f619518cc8dab388d29156d89534daed61c43bcede8b0e WatchSource:0}: Error finding container 16dd09827f583dc0c3f619518cc8dab388d29156d89534daed61c43bcede8b0e: Status 404 returned error can't find the container with id 16dd09827f583dc0c3f619518cc8dab388d29156d89534daed61c43bcede8b0e Nov 03 08:45:58.173552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.173513 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:45:58.173608 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.173588 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:45:58.173639 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.173617 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:45:58.173755 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.173736 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tjz\" (UniqueName: \"kubernetes.io/projected/8fffcee0-fbee-424d-b719-31f2c91cd1d3-kube-api-access-m9tjz\") pod \"authorino-57f7f8659f-v8d5x\" (UID: \"8fffcee0-fbee-424d-b719-31f2c91cd1d3\") " pod="kuadrant-system/authorino-57f7f8659f-v8d5x" Nov 03 08:45:58.234000 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.233932 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9tjz\" (UniqueName: \"kubernetes.io/projected/8fffcee0-fbee-424d-b719-31f2c91cd1d3-kube-api-access-m9tjz\") pod \"authorino-57f7f8659f-v8d5x\" (UID: \"8fffcee0-fbee-424d-b719-31f2c91cd1d3\") " pod="kuadrant-system/authorino-57f7f8659f-v8d5x" Nov 03 08:45:58.313538 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.313502 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-v8d5x" Nov 03 08:45:58.420140 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.420114 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 08:45:58.478365 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.478325 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" event={"ID":"af4db30f-c036-4273-b29e-733cd2dbe65f","Type":"ContainerStarted","Data":"bc84daf272c55d1fab92c8ff6435442d567445b5a68efe112f15a40adea24562"} Nov 03 08:45:58.478550 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.478399 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" event={"ID":"af4db30f-c036-4273-b29e-733cd2dbe65f","Type":"ContainerStarted","Data":"16dd09827f583dc0c3f619518cc8dab388d29156d89534daed61c43bcede8b0e"} Nov 03 08:45:58.663504 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.663387 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" podStartSLOduration=1.663372877 podStartE2EDuration="1.663372877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:58.663149444 +0000 UTC m=+391.588091686" watchObservedRunningTime="2026-11-03 08:45:58.663372877 +0000 UTC m=+391.588315094" Nov 03 08:45:58.686993 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.686969 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-v8d5x"] Nov 03 08:45:58.689049 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:45:58.689018 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fffcee0_fbee_424d_b719_31f2c91cd1d3.slice/crio-66bccc4caf6cbc827f8f585ba0b7d81cc07b323158302687c30d07e887339ac2 WatchSource:0}: Error finding container 66bccc4caf6cbc827f8f585ba0b7d81cc07b323158302687c30d07e887339ac2: Status 404 returned error can't find the container with id 66bccc4caf6cbc827f8f585ba0b7d81cc07b323158302687c30d07e887339ac2 Nov 03 08:45:58.723808 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:58.723773 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:45:59.486119 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:59.486076 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-v8d5x" event={"ID":"8fffcee0-fbee-424d-b719-31f2c91cd1d3","Type":"ContainerStarted","Data":"66bccc4caf6cbc827f8f585ba0b7d81cc07b323158302687c30d07e887339ac2"} Nov 03 08:45:59.729240 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:45:59.729211 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:46:00.494637 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:00.494591 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" event={"ID":"a3dd0096-28c3-4017-9056-54b721df8abe","Type":"ContainerStarted","Data":"9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26"} Nov 03 08:46:00.495141 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:00.494879 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:46:00.495141 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:00.494908 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:46:00.495466 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:00.495439 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-86rlh" Nov 03 08:46:00.614093 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:00.614038 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" podStartSLOduration=1.818252352 podStartE2EDuration="4.614024255s" totalImagesPullingTime="2.795771903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:00.612903672 +0000 UTC m=+393.537845908" watchObservedRunningTime="2026-11-03 08:46:00.614024255 +0000 UTC m=+393.538966485" Nov 03 08:46:02.507242 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:02.507203 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-v8d5x" event={"ID":"8fffcee0-fbee-424d-b719-31f2c91cd1d3","Type":"ContainerStarted","Data":"31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b"} Nov 03 08:46:02.623554 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:02.623500 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-v8d5x" podStartSLOduration=2.752485841 podStartE2EDuration="5.623461839s" totalImagesPullingTime="2.870975998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:02.62181666 +0000 UTC m=+395.546758895" watchObservedRunningTime="2026-11-03 08:46:02.623461839 +0000 UTC m=+395.548404056" Nov 03 08:46:04.207380 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:04.207343 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-v8d5x"] Nov 03 08:46:04.517884 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:04.517841 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-v8d5x" podUID="8fffcee0-fbee-424d-b719-31f2c91cd1d3" containerName="authorino" containerID="cri-o://31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b" gracePeriod=30 Nov 03 08:46:04.793663 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:04.793638 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-v8d5x" Nov 03 08:46:04.937752 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:04.937713 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fffcee0-fbee-424d-b719-31f2c91cd1d3-kube-api-access-m9tjz\" (UniqueName: \"kubernetes.io/projected/8fffcee0-fbee-424d-b719-31f2c91cd1d3-kube-api-access-m9tjz\") pod \"8fffcee0-fbee-424d-b719-31f2c91cd1d3\" (UID: \"8fffcee0-fbee-424d-b719-31f2c91cd1d3\") " Nov 03 08:46:04.940305 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:04.940281 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fffcee0-fbee-424d-b719-31f2c91cd1d3-kube-api-access-m9tjz" pod "8fffcee0-fbee-424d-b719-31f2c91cd1d3" (UID: "8fffcee0-fbee-424d-b719-31f2c91cd1d3"). InnerVolumeSpecName "kube-api-access-m9tjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:46:05.038789 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:05.038703 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9tjz\" (UniqueName: \"kubernetes.io/projected/8fffcee0-fbee-424d-b719-31f2c91cd1d3-kube-api-access-m9tjz\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:46:05.523886 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:05.523845 5015 generic.go:356] "Generic (PLEG): container finished" podID="8fffcee0-fbee-424d-b719-31f2c91cd1d3" containerID="31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b" exitCode=0 Nov 03 08:46:05.523886 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:05.523886 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-v8d5x" Nov 03 08:46:05.524431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:05.523927 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-v8d5x" event={"ID":"8fffcee0-fbee-424d-b719-31f2c91cd1d3","Type":"ContainerDied","Data":"31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b"} Nov 03 08:46:05.524431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:05.523971 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-v8d5x" event={"ID":"8fffcee0-fbee-424d-b719-31f2c91cd1d3","Type":"ContainerDied","Data":"66bccc4caf6cbc827f8f585ba0b7d81cc07b323158302687c30d07e887339ac2"} Nov 03 08:46:05.524431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:05.523988 5015 scope.go:122] "RemoveContainer" containerID="31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b" Nov 03 08:46:05.533108 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:05.533091 5015 scope.go:122] "RemoveContainer" containerID="31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b" Nov 03 08:46:05.533379 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:46:05.533356 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b\": container with ID starting with 31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b not found: ID does not exist" containerID="31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b" Nov 03 08:46:05.533445 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:05.533390 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b"} err="failed to get container status \"31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b\": rpc error: code = NotFound desc = could not find container \"31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b\": container with ID starting with 31bacd491aa99a93e2b9f3e86bed60764fd0d66c82ee606a3f0b2fc83f6a2e9b not found: ID does not exist" Nov 03 08:46:05.694739 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:05.694701 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-v8d5x"] Nov 03 08:46:05.700597 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:05.700566 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-v8d5x"] Nov 03 08:46:07.203033 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:07.203002 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fffcee0-fbee-424d-b719-31f2c91cd1d3" path="/var/lib/kubelet/pods/8fffcee0-fbee-424d-b719-31f2c91cd1d3/volumes" Nov 03 08:46:11.522128 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.522090 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd"] Nov 03 08:46:11.522531 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.522492 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8fffcee0-fbee-424d-b719-31f2c91cd1d3" containerName="authorino" Nov 03 08:46:11.522531 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.522504 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fffcee0-fbee-424d-b719-31f2c91cd1d3" containerName="authorino" Nov 03 08:46:11.522621 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.522596 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fffcee0-fbee-424d-b719-31f2c91cd1d3" containerName="authorino" Nov 03 08:46:11.525973 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.525956 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.580858 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.580819 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 08:46:11.581031 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.580876 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 08:46:11.598299 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.598273 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e6a48fda-d04e-423f-89dc-a640c4b2d513-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.598402 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.598326 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxtxh\" (UniqueName: \"kubernetes.io/projected/e6a48fda-d04e-423f-89dc-a640c4b2d513-kube-api-access-cxtxh\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.598402 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.598386 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6a48fda-d04e-423f-89dc-a640c4b2d513-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.598471 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.598435 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6a48fda-d04e-423f-89dc-a640c4b2d513-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.680957 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.680910 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-p9w5g\"" Nov 03 08:46:11.699619 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.699585 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e6a48fda-d04e-423f-89dc-a640c4b2d513-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.699781 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.699640 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxtxh\" (UniqueName: \"kubernetes.io/projected/e6a48fda-d04e-423f-89dc-a640c4b2d513-kube-api-access-cxtxh\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.699781 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.699666 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6a48fda-d04e-423f-89dc-a640c4b2d513-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.699781 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.699710 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6a48fda-d04e-423f-89dc-a640c4b2d513-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.699993 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.699974 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e6a48fda-d04e-423f-89dc-a640c4b2d513-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.700030 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.699996 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6a48fda-d04e-423f-89dc-a640c4b2d513-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.731841 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.731804 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 08:46:11.734144 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.734122 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd"] Nov 03 08:46:11.741449 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.741421 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6a48fda-d04e-423f-89dc-a640c4b2d513-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.759617 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.759588 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxtxh\" (UniqueName: \"kubernetes.io/projected/e6a48fda-d04e-423f-89dc-a640c4b2d513-kube-api-access-cxtxh\") pod \"kserve-module-controller-manager-7566cf5fc8-wwbxd\" (UID: \"e6a48fda-d04e-423f-89dc-a640c4b2d513\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:11.838813 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:11.838728 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:12.266087 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:12.266059 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd"] Nov 03 08:46:12.267624 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:46:12.267599 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a48fda_d04e_423f_89dc_a640c4b2d513.slice/crio-73ab0507a27b8c414438cf8d8888e79c2a4337acfddff1862cdc414831e29235 WatchSource:0}: Error finding container 73ab0507a27b8c414438cf8d8888e79c2a4337acfddff1862cdc414831e29235: Status 404 returned error can't find the container with id 73ab0507a27b8c414438cf8d8888e79c2a4337acfddff1862cdc414831e29235 Nov 03 08:46:12.508853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:12.508817 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:46:12.564992 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:12.564910 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" event={"ID":"e6a48fda-d04e-423f-89dc-a640c4b2d513","Type":"ContainerStarted","Data":"73ab0507a27b8c414438cf8d8888e79c2a4337acfddff1862cdc414831e29235"} Nov 03 08:46:15.584968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:15.584927 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" event={"ID":"e6a48fda-d04e-423f-89dc-a640c4b2d513","Type":"ContainerStarted","Data":"e2cd6ddb6698ac1cb9d0645465bfa3b667517feb6b5b4cbb668afb3029b729fc"} Nov 03 08:46:15.585384 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:15.585031 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:15.754366 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:15.754303 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" podStartSLOduration=1.952463987 podStartE2EDuration="4.754285859s" totalImagesPullingTime="2.801821872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:15.753983151 +0000 UTC m=+408.678925364" watchObservedRunningTime="2026-11-03 08:46:15.754285859 +0000 UTC m=+408.679228077" Nov 03 08:46:16.588610 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:16.588570 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zwxdh"] Nov 03 08:46:16.589021 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:16.588794 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" podUID="a3dd0096-28c3-4017-9056-54b721df8abe" containerName="limitador" containerID="cri-o://9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26" gracePeriod=30 Nov 03 08:46:16.910053 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:46:16.910018 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:46:17.563011 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.562984 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:46:17.597155 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.597121 5015 generic.go:356] "Generic (PLEG): container finished" podID="a3dd0096-28c3-4017-9056-54b721df8abe" containerID="9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26" exitCode=0 Nov 03 08:46:17.597675 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.597168 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" Nov 03 08:46:17.597675 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.597210 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" event={"ID":"a3dd0096-28c3-4017-9056-54b721df8abe","Type":"ContainerDied","Data":"9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26"} Nov 03 08:46:17.597675 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.597280 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zwxdh" event={"ID":"a3dd0096-28c3-4017-9056-54b721df8abe","Type":"ContainerDied","Data":"5a5f569a47d2be0761b8d127806143f5bfe1aa06680f19443c4406f745e07da5"} Nov 03 08:46:17.597675 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.597314 5015 scope.go:122] "RemoveContainer" containerID="9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26" Nov 03 08:46:17.606456 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.606429 5015 scope.go:122] "RemoveContainer" containerID="9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26" Nov 03 08:46:17.606755 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:46:17.606736 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26\": container with ID starting with 9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26 not found: ID does not exist" containerID="9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26" Nov 03 08:46:17.606816 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.606768 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26"} err="failed to get container status \"9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26\": rpc error: code = NotFound desc = could not find container \"9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26\": container with ID starting with 9e9ad8a1498a7a365e4c039e645062a921d1bfd4cf77c4b312235e041db80f26 not found: ID does not exist" Nov 03 08:46:17.660643 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.660612 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dd0096-28c3-4017-9056-54b721df8abe-kube-api-access-5h5fg\" (UniqueName: \"kubernetes.io/projected/a3dd0096-28c3-4017-9056-54b721df8abe-kube-api-access-5h5fg\") pod \"a3dd0096-28c3-4017-9056-54b721df8abe\" (UID: \"a3dd0096-28c3-4017-9056-54b721df8abe\") " Nov 03 08:46:17.660643 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.660651 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3dd0096-28c3-4017-9056-54b721df8abe-config-file\" (UniqueName: \"kubernetes.io/configmap/a3dd0096-28c3-4017-9056-54b721df8abe-config-file\") pod \"a3dd0096-28c3-4017-9056-54b721df8abe\" (UID: \"a3dd0096-28c3-4017-9056-54b721df8abe\") " Nov 03 08:46:17.661089 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.661028 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3dd0096-28c3-4017-9056-54b721df8abe-config-file" pod "a3dd0096-28c3-4017-9056-54b721df8abe" (UID: "a3dd0096-28c3-4017-9056-54b721df8abe"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:46:17.662907 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.662889 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dd0096-28c3-4017-9056-54b721df8abe-kube-api-access-5h5fg" pod "a3dd0096-28c3-4017-9056-54b721df8abe" (UID: "a3dd0096-28c3-4017-9056-54b721df8abe"). InnerVolumeSpecName "kube-api-access-5h5fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:46:17.762308 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.762273 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h5fg\" (UniqueName: \"kubernetes.io/projected/a3dd0096-28c3-4017-9056-54b721df8abe-kube-api-access-5h5fg\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:46:17.762308 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:17.762303 5015 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a3dd0096-28c3-4017-9056-54b721df8abe-config-file\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:46:18.021594 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:18.021558 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zwxdh"] Nov 03 08:46:18.026399 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:18.026376 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zwxdh"] Nov 03 08:46:19.203319 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:19.203281 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3dd0096-28c3-4017-9056-54b721df8abe" path="/var/lib/kubelet/pods/a3dd0096-28c3-4017-9056-54b721df8abe/volumes" Nov 03 08:46:22.398569 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.398535 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p"] Nov 03 08:46:22.398959 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.398897 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a3dd0096-28c3-4017-9056-54b721df8abe" containerName="limitador" Nov 03 08:46:22.398959 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.398907 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dd0096-28c3-4017-9056-54b721df8abe" containerName="limitador" Nov 03 08:46:22.399032 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.398969 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dd0096-28c3-4017-9056-54b721df8abe" containerName="limitador" Nov 03 08:46:22.401602 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.401586 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" Nov 03 08:46:22.459221 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.459194 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p"] Nov 03 08:46:22.504986 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.504954 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bhsf\" (UniqueName: \"kubernetes.io/projected/4757c90e-41ba-46ed-8388-7f8c489d4b86-kube-api-access-9bhsf\") pod \"postgres-7cdbf6dc44-lf65p\" (UID: \"4757c90e-41ba-46ed-8388-7f8c489d4b86\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" Nov 03 08:46:22.505143 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.504997 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4757c90e-41ba-46ed-8388-7f8c489d4b86-data\") pod \"postgres-7cdbf6dc44-lf65p\" (UID: \"4757c90e-41ba-46ed-8388-7f8c489d4b86\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" Nov 03 08:46:22.506869 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.506835 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 08:46:22.506981 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.506913 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 08:46:22.506981 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.506931 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 08:46:22.506981 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.506974 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-znxr8\"" Nov 03 08:46:22.606301 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.606267 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhsf\" (UniqueName: \"kubernetes.io/projected/4757c90e-41ba-46ed-8388-7f8c489d4b86-kube-api-access-9bhsf\") pod \"postgres-7cdbf6dc44-lf65p\" (UID: \"4757c90e-41ba-46ed-8388-7f8c489d4b86\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" Nov 03 08:46:22.606301 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.606299 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4757c90e-41ba-46ed-8388-7f8c489d4b86-data\") pod \"postgres-7cdbf6dc44-lf65p\" (UID: \"4757c90e-41ba-46ed-8388-7f8c489d4b86\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" Nov 03 08:46:22.606739 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.606723 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4757c90e-41ba-46ed-8388-7f8c489d4b86-data\") pod \"postgres-7cdbf6dc44-lf65p\" (UID: \"4757c90e-41ba-46ed-8388-7f8c489d4b86\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" Nov 03 08:46:22.665853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.665786 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhsf\" (UniqueName: \"kubernetes.io/projected/4757c90e-41ba-46ed-8388-7f8c489d4b86-kube-api-access-9bhsf\") pod \"postgres-7cdbf6dc44-lf65p\" (UID: \"4757c90e-41ba-46ed-8388-7f8c489d4b86\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" Nov 03 08:46:22.712662 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:22.712634 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" Nov 03 08:46:23.089870 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:23.089848 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p"] Nov 03 08:46:23.091956 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:46:23.091930 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4757c90e_41ba_46ed_8388_7f8c489d4b86.slice/crio-9a02bb4e13e03275bc7a76aa3a21c1b59b4cdebf0a4596035ca96bf4e86e0efd WatchSource:0}: Error finding container 9a02bb4e13e03275bc7a76aa3a21c1b59b4cdebf0a4596035ca96bf4e86e0efd: Status 404 returned error can't find the container with id 9a02bb4e13e03275bc7a76aa3a21c1b59b4cdebf0a4596035ca96bf4e86e0efd Nov 03 08:46:23.629328 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:23.629292 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" event={"ID":"4757c90e-41ba-46ed-8388-7f8c489d4b86","Type":"ContainerStarted","Data":"9a02bb4e13e03275bc7a76aa3a21c1b59b4cdebf0a4596035ca96bf4e86e0efd"} Nov 03 08:46:29.245137 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:29.245106 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 08:46:29.665821 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:29.665782 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" event={"ID":"4757c90e-41ba-46ed-8388-7f8c489d4b86","Type":"ContainerStarted","Data":"b18377b99854730ed68a2504dfaacdf323df95896b8b95e48d16752401b70f1a"} Nov 03 08:46:29.665998 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:29.665922 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" Nov 03 08:46:29.784879 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:29.784810 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" podStartSLOduration=1.685255398 podStartE2EDuration="7.784787711s" totalImagesPullingTime="6.099532313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:29.782144539 +0000 UTC m=+422.707086767" watchObservedRunningTime="2026-11-03 08:46:29.784787711 +0000 UTC m=+422.709729929" Nov 03 08:46:33.917087 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:33.917058 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q"] Nov 03 08:46:33.920760 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:33.920741 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:33.978720 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:33.978687 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q"] Nov 03 08:46:34.000820 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.000790 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/250a2b15-6da6-45dc-86fa-55bbe5752688-manager-config\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.000958 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.000829 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250a2b15-6da6-45dc-86fa-55bbe5752688-cert\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.000958 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.000864 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ktjm\" (UniqueName: \"kubernetes.io/projected/250a2b15-6da6-45dc-86fa-55bbe5752688-kube-api-access-4ktjm\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.001027 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.000995 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/250a2b15-6da6-45dc-86fa-55bbe5752688-metrics-certs\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.026658 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.026621 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 08:46:34.026869 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.026851 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:46:34.026942 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.026861 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-95l6p\"" Nov 03 08:46:34.026942 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.026876 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 08:46:34.026942 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.026894 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 08:46:34.026942 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.026901 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 08:46:34.101663 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.101628 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/250a2b15-6da6-45dc-86fa-55bbe5752688-manager-config\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.101833 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.101676 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250a2b15-6da6-45dc-86fa-55bbe5752688-cert\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.101833 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.101727 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktjm\" (UniqueName: \"kubernetes.io/projected/250a2b15-6da6-45dc-86fa-55bbe5752688-kube-api-access-4ktjm\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.101833 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.101767 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/250a2b15-6da6-45dc-86fa-55bbe5752688-metrics-certs\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.102532 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.102511 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/250a2b15-6da6-45dc-86fa-55bbe5752688-manager-config\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.104398 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.104376 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250a2b15-6da6-45dc-86fa-55bbe5752688-cert\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.104507 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.104415 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/250a2b15-6da6-45dc-86fa-55bbe5752688-metrics-certs\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.161628 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.161596 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktjm\" (UniqueName: \"kubernetes.io/projected/250a2b15-6da6-45dc-86fa-55bbe5752688-kube-api-access-4ktjm\") pod \"jobset-controller-manager-74879684bf-qbl6q\" (UID: \"250a2b15-6da6-45dc-86fa-55bbe5752688\") " pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.233858 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.233830 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:34.603887 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.603861 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q"] Nov 03 08:46:34.605399 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:46:34.605376 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250a2b15_6da6_45dc_86fa_55bbe5752688.slice/crio-b0afe07a25ab30e67941ad4ebbf5a19354550bd310f806576def2dfd9427bb2f WatchSource:0}: Error finding container b0afe07a25ab30e67941ad4ebbf5a19354550bd310f806576def2dfd9427bb2f: Status 404 returned error can't find the container with id b0afe07a25ab30e67941ad4ebbf5a19354550bd310f806576def2dfd9427bb2f Nov 03 08:46:34.693895 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:34.693859 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" event={"ID":"250a2b15-6da6-45dc-86fa-55bbe5752688","Type":"ContainerStarted","Data":"b0afe07a25ab30e67941ad4ebbf5a19354550bd310f806576def2dfd9427bb2f"} Nov 03 08:46:35.700429 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:35.700394 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lf65p" Nov 03 08:46:36.591328 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:36.591300 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wwbxd" Nov 03 08:46:38.718089 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:38.718049 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" event={"ID":"250a2b15-6da6-45dc-86fa-55bbe5752688","Type":"ContainerStarted","Data":"16b124a6293f131f81ffb1a0027ae9d87abde4105f22ae53b9f648c79553bac0"} Nov 03 08:46:38.718443 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:38.718139 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:46:38.837447 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:38.837392 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" podStartSLOduration=1.8251544640000001 podStartE2EDuration="5.837378078s" totalImagesPullingTime="4.012223614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:38.835614916 +0000 UTC m=+431.760557153" watchObservedRunningTime="2026-11-03 08:46:38.837378078 +0000 UTC m=+431.762320294" Nov 03 08:46:46.901286 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:46:46.901249 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:46:49.727928 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:46:49.727896 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-74879684bf-qbl6q" Nov 03 08:47:16.903799 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:47:16.903756 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:47:46.832704 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:46.832656 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c89vt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:47:46.833221 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:46.832733 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:47:46.903118 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:47:46.903069 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:47:48.202157 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.202114 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xx84n"] Nov 03 08:47:48.206859 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.206832 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xx84n" Nov 03 08:47:48.233365 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.233334 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcs74\" (UniqueName: \"kubernetes.io/projected/5a1b60fd-a83a-4eaf-9950-94398589b3f1-kube-api-access-vcs74\") pod \"authorino-6f6944bcdd-xx84n\" (UID: \"5a1b60fd-a83a-4eaf-9950-94398589b3f1\") " pod="kuadrant-system/authorino-6f6944bcdd-xx84n" Nov 03 08:47:48.262989 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.262953 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xx84n"] Nov 03 08:47:48.309588 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.309559 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-h722q\"" Nov 03 08:47:48.334710 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.334683 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcs74\" (UniqueName: \"kubernetes.io/projected/5a1b60fd-a83a-4eaf-9950-94398589b3f1-kube-api-access-vcs74\") pod \"authorino-6f6944bcdd-xx84n\" (UID: \"5a1b60fd-a83a-4eaf-9950-94398589b3f1\") " pod="kuadrant-system/authorino-6f6944bcdd-xx84n" Nov 03 08:47:48.395109 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.395071 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcs74\" (UniqueName: \"kubernetes.io/projected/5a1b60fd-a83a-4eaf-9950-94398589b3f1-kube-api-access-vcs74\") pod \"authorino-6f6944bcdd-xx84n\" (UID: \"5a1b60fd-a83a-4eaf-9950-94398589b3f1\") " pod="kuadrant-system/authorino-6f6944bcdd-xx84n" Nov 03 08:47:48.516801 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.516768 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xx84n"] Nov 03 08:47:48.516996 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.516985 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xx84n" Nov 03 08:47:48.542495 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.542443 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f5675bbd5-slmjp"] Nov 03 08:47:48.546204 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.546184 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5675bbd5-slmjp" Nov 03 08:47:48.605103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.605070 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f5675bbd5-slmjp"] Nov 03 08:47:48.637606 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.637576 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n652\" (UniqueName: \"kubernetes.io/projected/d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310-kube-api-access-5n652\") pod \"authorino-5f5675bbd5-slmjp\" (UID: \"d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310\") " pod="kuadrant-system/authorino-5f5675bbd5-slmjp" Nov 03 08:47:48.738544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.738507 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n652\" (UniqueName: \"kubernetes.io/projected/d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310-kube-api-access-5n652\") pod \"authorino-5f5675bbd5-slmjp\" (UID: \"d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310\") " pod="kuadrant-system/authorino-5f5675bbd5-slmjp" Nov 03 08:47:48.797851 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.797789 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n652\" (UniqueName: \"kubernetes.io/projected/d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310-kube-api-access-5n652\") pod \"authorino-5f5675bbd5-slmjp\" (UID: \"d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310\") " pod="kuadrant-system/authorino-5f5675bbd5-slmjp" Nov 03 08:47:48.859377 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.859342 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5675bbd5-slmjp" Nov 03 08:47:48.899188 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:48.899152 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xx84n"] Nov 03 08:47:48.902342 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:47:48.902313 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a1b60fd_a83a_4eaf_9950_94398589b3f1.slice/crio-cd7efa898a1102722f3ffe8f6427c104534678ff23098621cd3ce49dc33019eb WatchSource:0}: Error finding container cd7efa898a1102722f3ffe8f6427c104534678ff23098621cd3ce49dc33019eb: Status 404 returned error can't find the container with id cd7efa898a1102722f3ffe8f6427c104534678ff23098621cd3ce49dc33019eb Nov 03 08:47:49.041923 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.041884 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f5675bbd5-slmjp"] Nov 03 08:47:49.074829 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.074754 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cfccc9779-wgvs5"] Nov 03 08:47:49.077902 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.077882 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cfccc9779-wgvs5" Nov 03 08:47:49.107940 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.107912 5015 generic.go:356] "Generic (PLEG): container finished" podID="66f961ac-69cf-407a-af9b-07611b20726c" containerID="ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781" exitCode=0 Nov 03 08:47:49.108086 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.107983 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" event={"ID":"66f961ac-69cf-407a-af9b-07611b20726c","Type":"ContainerDied","Data":"ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781"} Nov 03 08:47:49.108375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.108354 5015 scope.go:122] "RemoveContainer" containerID="ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781" Nov 03 08:47:49.109196 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.109171 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-xx84n" event={"ID":"5a1b60fd-a83a-4eaf-9950-94398589b3f1","Type":"ContainerStarted","Data":"cd7efa898a1102722f3ffe8f6427c104534678ff23098621cd3ce49dc33019eb"} Nov 03 08:47:49.109576 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.109549 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:47:49.109645 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.109615 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:47:49.109682 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.109647 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:47:49.134792 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.134762 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 08:47:49.136267 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.136240 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cfccc9779-wgvs5"] Nov 03 08:47:49.142687 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.142665 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdpqh\" (UniqueName: \"kubernetes.io/projected/96182b4d-0fb2-4819-ba44-b4a3e069548d-kube-api-access-cdpqh\") pod \"authorino-cfccc9779-wgvs5\" (UID: \"96182b4d-0fb2-4819-ba44-b4a3e069548d\") " pod="kuadrant-system/authorino-cfccc9779-wgvs5" Nov 03 08:47:49.142810 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.142733 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96182b4d-0fb2-4819-ba44-b4a3e069548d-tls-cert\") pod \"authorino-cfccc9779-wgvs5\" (UID: \"96182b4d-0fb2-4819-ba44-b4a3e069548d\") " pod="kuadrant-system/authorino-cfccc9779-wgvs5" Nov 03 08:47:49.231377 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.231353 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f5675bbd5-slmjp"] Nov 03 08:47:49.233715 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:47:49.233689 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7b07e0e_7b5c_4c21_8bf1_3ff4dc711310.slice/crio-8f7a737ed5d9e1ae4a4510e08257c6d0dce38c0e3af4b3fabf4f2bc05ba58c4d WatchSource:0}: Error finding container 8f7a737ed5d9e1ae4a4510e08257c6d0dce38c0e3af4b3fabf4f2bc05ba58c4d: Status 404 returned error can't find the container with id 8f7a737ed5d9e1ae4a4510e08257c6d0dce38c0e3af4b3fabf4f2bc05ba58c4d Nov 03 08:47:49.243762 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.243732 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdpqh\" (UniqueName: \"kubernetes.io/projected/96182b4d-0fb2-4819-ba44-b4a3e069548d-kube-api-access-cdpqh\") pod \"authorino-cfccc9779-wgvs5\" (UID: \"96182b4d-0fb2-4819-ba44-b4a3e069548d\") " pod="kuadrant-system/authorino-cfccc9779-wgvs5" Nov 03 08:47:49.243882 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.243782 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96182b4d-0fb2-4819-ba44-b4a3e069548d-tls-cert\") pod \"authorino-cfccc9779-wgvs5\" (UID: \"96182b4d-0fb2-4819-ba44-b4a3e069548d\") " pod="kuadrant-system/authorino-cfccc9779-wgvs5" Nov 03 08:47:49.246278 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.246252 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96182b4d-0fb2-4819-ba44-b4a3e069548d-tls-cert\") pod \"authorino-cfccc9779-wgvs5\" (UID: \"96182b4d-0fb2-4819-ba44-b4a3e069548d\") " pod="kuadrant-system/authorino-cfccc9779-wgvs5" Nov 03 08:47:49.303959 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.303923 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdpqh\" (UniqueName: \"kubernetes.io/projected/96182b4d-0fb2-4819-ba44-b4a3e069548d-kube-api-access-cdpqh\") pod \"authorino-cfccc9779-wgvs5\" (UID: \"96182b4d-0fb2-4819-ba44-b4a3e069548d\") " pod="kuadrant-system/authorino-cfccc9779-wgvs5" Nov 03 08:47:49.388460 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.388369 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cfccc9779-wgvs5" Nov 03 08:47:49.779106 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:49.779080 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cfccc9779-wgvs5"] Nov 03 08:47:49.780554 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:47:49.780521 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96182b4d_0fb2_4819_ba44_b4a3e069548d.slice/crio-16a2f0a4df2be0c94e168edd73fa93581a3cb4bf10e3da6598339ba095916c86 WatchSource:0}: Error finding container 16a2f0a4df2be0c94e168edd73fa93581a3cb4bf10e3da6598339ba095916c86: Status 404 returned error can't find the container with id 16a2f0a4df2be0c94e168edd73fa93581a3cb4bf10e3da6598339ba095916c86 Nov 03 08:47:50.115880 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:50.115785 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cfccc9779-wgvs5" event={"ID":"96182b4d-0fb2-4819-ba44-b4a3e069548d","Type":"ContainerStarted","Data":"16a2f0a4df2be0c94e168edd73fa93581a3cb4bf10e3da6598339ba095916c86"} Nov 03 08:47:50.117791 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:50.117757 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" event={"ID":"66f961ac-69cf-407a-af9b-07611b20726c","Type":"ContainerStarted","Data":"f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c"} Nov 03 08:47:50.118996 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:50.118970 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5675bbd5-slmjp" event={"ID":"d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310","Type":"ContainerStarted","Data":"8f7a737ed5d9e1ae4a4510e08257c6d0dce38c0e3af4b3fabf4f2bc05ba58c4d"} Nov 03 08:47:50.261908 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:50.261819 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:47:50.262417 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:50.262080 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c89vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 08:47:50.262417 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:50.262128 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 08:47:51.127812 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.127754 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5675bbd5-slmjp" event={"ID":"d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310","Type":"ContainerStarted","Data":"5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825"} Nov 03 08:47:51.128035 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.127788 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5f5675bbd5-slmjp" podUID="d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310" containerName="authorino" containerID="cri-o://5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825" gracePeriod=30 Nov 03 08:47:51.130095 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.130069 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-xx84n" event={"ID":"5a1b60fd-a83a-4eaf-9950-94398589b3f1","Type":"ContainerStarted","Data":"c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a"} Nov 03 08:47:51.130201 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.130152 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-xx84n" podUID="5a1b60fd-a83a-4eaf-9950-94398589b3f1" containerName="authorino" containerID="cri-o://c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a" gracePeriod=30 Nov 03 08:47:51.258502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.253339 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c89vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 08:47:51.258502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.253404 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 08:47:51.266324 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.266272 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f5675bbd5-slmjp" podStartSLOduration=1.906603397 podStartE2EDuration="3.266258891s" totalImagesPullingTime="1.359655494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:51.252822459 +0000 UTC m=+504.177764708" watchObservedRunningTime="2026-11-03 08:47:51.266258891 +0000 UTC m=+504.191201107" Nov 03 08:47:51.370551 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.370463 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-xx84n" podStartSLOduration=1.799042901 podStartE2EDuration="3.370445083s" totalImagesPullingTime="1.571402182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:51.369820296 +0000 UTC m=+504.294762515" watchObservedRunningTime="2026-11-03 08:47:51.370445083 +0000 UTC m=+504.295387327" Nov 03 08:47:51.476147 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.476126 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xx84n" Nov 03 08:47:51.479125 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.479102 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5675bbd5-slmjp" Nov 03 08:47:51.577992 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.577957 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1b60fd-a83a-4eaf-9950-94398589b3f1-kube-api-access-vcs74\" (UniqueName: \"kubernetes.io/projected/5a1b60fd-a83a-4eaf-9950-94398589b3f1-kube-api-access-vcs74\") pod \"5a1b60fd-a83a-4eaf-9950-94398589b3f1\" (UID: \"5a1b60fd-a83a-4eaf-9950-94398589b3f1\") " Nov 03 08:47:51.578160 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.578032 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310-kube-api-access-5n652\" (UniqueName: \"kubernetes.io/projected/d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310-kube-api-access-5n652\") pod \"d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310\" (UID: \"d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310\") " Nov 03 08:47:51.580323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.580294 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310-kube-api-access-5n652" pod "d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310" (UID: "d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310"). InnerVolumeSpecName "kube-api-access-5n652". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:47:51.580323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.580301 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1b60fd-a83a-4eaf-9950-94398589b3f1-kube-api-access-vcs74" pod "5a1b60fd-a83a-4eaf-9950-94398589b3f1" (UID: "5a1b60fd-a83a-4eaf-9950-94398589b3f1"). InnerVolumeSpecName "kube-api-access-vcs74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:47:51.679303 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.679216 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n652\" (UniqueName: \"kubernetes.io/projected/d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310-kube-api-access-5n652\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:47:51.679303 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:51.679246 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcs74\" (UniqueName: \"kubernetes.io/projected/5a1b60fd-a83a-4eaf-9950-94398589b3f1-kube-api-access-vcs74\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:47:52.162985 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.162944 5015 generic.go:356] "Generic (PLEG): container finished" podID="5a1b60fd-a83a-4eaf-9950-94398589b3f1" containerID="c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a" exitCode=0 Nov 03 08:47:52.162985 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.162987 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xx84n" Nov 03 08:47:52.163253 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.163030 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-xx84n" event={"ID":"5a1b60fd-a83a-4eaf-9950-94398589b3f1","Type":"ContainerDied","Data":"c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a"} Nov 03 08:47:52.163253 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.163084 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-xx84n" event={"ID":"5a1b60fd-a83a-4eaf-9950-94398589b3f1","Type":"ContainerDied","Data":"cd7efa898a1102722f3ffe8f6427c104534678ff23098621cd3ce49dc33019eb"} Nov 03 08:47:52.163253 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.163107 5015 scope.go:122] "RemoveContainer" containerID="c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a" Nov 03 08:47:52.165303 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.165274 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cfccc9779-wgvs5" event={"ID":"96182b4d-0fb2-4819-ba44-b4a3e069548d","Type":"ContainerStarted","Data":"9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79"} Nov 03 08:47:52.167053 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.167026 5015 generic.go:356] "Generic (PLEG): container finished" podID="d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310" containerID="5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825" exitCode=0 Nov 03 08:47:52.167186 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.167068 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5675bbd5-slmjp" Nov 03 08:47:52.167186 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.167119 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5675bbd5-slmjp" event={"ID":"d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310","Type":"ContainerDied","Data":"5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825"} Nov 03 08:47:52.167186 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.167158 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5675bbd5-slmjp" event={"ID":"d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310","Type":"ContainerDied","Data":"8f7a737ed5d9e1ae4a4510e08257c6d0dce38c0e3af4b3fabf4f2bc05ba58c4d"} Nov 03 08:47:52.174842 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.174820 5015 scope.go:122] "RemoveContainer" containerID="c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a" Nov 03 08:47:52.175161 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:47:52.175134 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a\": container with ID starting with c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a not found: ID does not exist" containerID="c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a" Nov 03 08:47:52.175317 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.175177 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a"} err="failed to get container status \"c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a\": rpc error: code = NotFound desc = could not find container \"c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a\": container with ID starting with c3b3ed1e51ecf4b9d66cda32c259d3f05318c7a0dd4f1e9292102b8456c23f8a not found: ID does not exist" Nov 03 08:47:52.175317 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.175202 5015 scope.go:122] "RemoveContainer" containerID="5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825" Nov 03 08:47:52.187752 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.187734 5015 scope.go:122] "RemoveContainer" containerID="5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825" Nov 03 08:47:52.188050 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:47:52.188032 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825\": container with ID starting with 5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825 not found: ID does not exist" containerID="5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825" Nov 03 08:47:52.188116 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.188059 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825"} err="failed to get container status \"5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825\": rpc error: code = NotFound desc = could not find container \"5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825\": container with ID starting with 5e13bdbf36733da8d74c86d5c24dfb95040ab32563ce9606c9318ad665b48825 not found: ID does not exist" Nov 03 08:47:52.247504 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.244081 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c89vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 08:47:52.247504 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.244140 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 08:47:52.331768 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.331730 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xx84n"] Nov 03 08:47:52.335734 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.335706 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xx84n"] Nov 03 08:47:52.455316 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.455191 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-cfccc9779-wgvs5" podStartSLOduration=1.957827749 podStartE2EDuration="3.455169386s" totalImagesPullingTime="1.497341637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:52.451873125 +0000 UTC m=+505.376815343" watchObservedRunningTime="2026-11-03 08:47:52.455169386 +0000 UTC m=+505.380111606" Nov 03 08:47:52.567516 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.566797 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f5675bbd5-slmjp"] Nov 03 08:47:52.573248 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:52.573211 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f5675bbd5-slmjp"] Nov 03 08:47:53.202972 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:53.202937 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a1b60fd-a83a-4eaf-9950-94398589b3f1" path="/var/lib/kubelet/pods/5a1b60fd-a83a-4eaf-9950-94398589b3f1/volumes" Nov 03 08:47:53.203329 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:53.203308 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310" path="/var/lib/kubelet/pods/d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310/volumes" Nov 03 08:47:53.261504 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:53.244844 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c89vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 08:47:53.261504 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:53.244912 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 08:47:54.250845 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:54.250808 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c89vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 08:47:54.251394 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:54.251362 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 08:47:55.245813 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:55.245784 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:47:55.246067 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:55.246047 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:47:55.246691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:47:55.246675 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 08:48:16.902205 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:48:16.902162 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:48:46.900871 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:48:46.900831 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:49:16.902863 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:49:16.902774 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:49:27.301370 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:27.301334 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 08:49:27.302068 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:27.302039 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 08:49:46.900900 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:49:46.900860 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:49:59.259650 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.259618 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd"] Nov 03 08:49:59.260143 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.260005 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5a1b60fd-a83a-4eaf-9950-94398589b3f1" containerName="authorino" Nov 03 08:49:59.260143 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.260016 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1b60fd-a83a-4eaf-9950-94398589b3f1" containerName="authorino" Nov 03 08:49:59.260143 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.260026 5015 cpu_manager.go:414] "containerMap: removing container" podUID="d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310" containerName="authorino" Nov 03 08:49:59.260143 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.260031 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310" containerName="authorino" Nov 03 08:49:59.260143 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.260094 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1b60fd-a83a-4eaf-9950-94398589b3f1" containerName="authorino" Nov 03 08:49:59.260143 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.260102 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b07e0e-7b5c-4c21-8bf1-3ff4dc711310" containerName="authorino" Nov 03 08:49:59.263254 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.263237 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.322960 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.322931 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd"] Nov 03 08:49:59.367545 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.367517 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 08:49:59.367545 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.367531 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 08:49:59.367726 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.367577 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7rrjj\"" Nov 03 08:49:59.367726 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.367650 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 08:49:59.399288 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.399241 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.399520 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.399337 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.399620 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.399517 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.399620 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.399569 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f925e33-6569-4ac4-9422-1ac38473de7a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.399620 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.399604 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.399786 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.399628 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg227\" (UniqueName: \"kubernetes.io/projected/6f925e33-6569-4ac4-9422-1ac38473de7a-kube-api-access-sg227\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.500093 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.500062 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.500093 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.500099 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.500331 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.500184 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.500331 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.500223 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f925e33-6569-4ac4-9422-1ac38473de7a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.500331 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.500250 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.500639 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.500276 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg227\" (UniqueName: \"kubernetes.io/projected/6f925e33-6569-4ac4-9422-1ac38473de7a-kube-api-access-sg227\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.500639 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.500610 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.500747 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.500655 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.500747 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.500540 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.502650 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.502627 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f925e33-6569-4ac4-9422-1ac38473de7a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.503065 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.503045 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f925e33-6569-4ac4-9422-1ac38473de7a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.560191 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.560116 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg227\" (UniqueName: \"kubernetes.io/projected/6f925e33-6569-4ac4-9422-1ac38473de7a-kube-api-access-sg227\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qrhwd\" (UID: \"6f925e33-6569-4ac4-9422-1ac38473de7a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.575640 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.575611 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:49:59.955691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.955659 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd"] Nov 03 08:49:59.957426 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:49:59.957395 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f925e33_6569_4ac4_9422_1ac38473de7a.slice/crio-20f205e22d973bb6f218e99d91cac9e940eed02a29e36110d1d53cb1aaf8d19a WatchSource:0}: Error finding container 20f205e22d973bb6f218e99d91cac9e940eed02a29e36110d1d53cb1aaf8d19a: Status 404 returned error can't find the container with id 20f205e22d973bb6f218e99d91cac9e940eed02a29e36110d1d53cb1aaf8d19a Nov 03 08:49:59.958667 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:49:59.958649 5015 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:50:00.871818 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:00.871782 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" event={"ID":"6f925e33-6569-4ac4-9422-1ac38473de7a","Type":"ContainerStarted","Data":"20f205e22d973bb6f218e99d91cac9e940eed02a29e36110d1d53cb1aaf8d19a"} Nov 03 08:50:02.889862 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:02.889822 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" event={"ID":"6f925e33-6569-4ac4-9422-1ac38473de7a","Type":"ContainerStarted","Data":"39441f55f4719d3e62bcf7b376c3a505b5f278e280eb21843b6949e8a14008ef"} Nov 03 08:50:03.011409 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:03.011347 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" podStartSLOduration=1.908991051 podStartE2EDuration="4.011333697s" totalImagesPullingTime="2.102342646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:03.00987204 +0000 UTC m=+635.934814257" watchObservedRunningTime="2026-11-03 08:50:03.011333697 +0000 UTC m=+635.936275917" Nov 03 08:50:03.575992 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:03.575954 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:50:08.123133 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.123102 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8"] Nov 03 08:50:08.126672 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.126653 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.181921 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.181883 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.182103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.181928 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pffth\" (UniqueName: \"kubernetes.io/projected/12c4a382-3942-493e-84a4-85d061fc87e4-kube-api-access-pffth\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.182103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.181999 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.182244 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.182132 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c4a382-3942-493e-84a4-85d061fc87e4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.182244 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.182181 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.182244 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.182207 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.230958 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.230925 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 08:50:08.233181 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.233157 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8"] Nov 03 08:50:08.283049 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.283011 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.283049 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.283052 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pffth\" (UniqueName: \"kubernetes.io/projected/12c4a382-3942-493e-84a4-85d061fc87e4-kube-api-access-pffth\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.283306 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.283074 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.283306 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.283120 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c4a382-3942-493e-84a4-85d061fc87e4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.283306 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.283146 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.283306 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.283162 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.283533 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.283504 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.283619 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.283600 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.283711 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.283688 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.285683 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.285660 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c4a382-3942-493e-84a4-85d061fc87e4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.285911 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.285893 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c4a382-3942-493e-84a4-85d061fc87e4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.349574 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.349547 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pffth\" (UniqueName: \"kubernetes.io/projected/12c4a382-3942-493e-84a4-85d061fc87e4-kube-api-access-pffth\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8\" (UID: \"12c4a382-3942-493e-84a4-85d061fc87e4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.439260 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.439175 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:08.818026 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.817995 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8"] Nov 03 08:50:08.819548 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:50:08.819522 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c4a382_3942_493e_84a4_85d061fc87e4.slice/crio-dd01b29ae097481485a3aea195eba23a71fc1d8a3f52d3f19c141a8374f98260 WatchSource:0}: Error finding container dd01b29ae097481485a3aea195eba23a71fc1d8a3f52d3f19c141a8374f98260: Status 404 returned error can't find the container with id dd01b29ae097481485a3aea195eba23a71fc1d8a3f52d3f19c141a8374f98260 Nov 03 08:50:08.931540 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:08.931507 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" event={"ID":"12c4a382-3942-493e-84a4-85d061fc87e4","Type":"ContainerStarted","Data":"dd01b29ae097481485a3aea195eba23a71fc1d8a3f52d3f19c141a8374f98260"} Nov 03 08:50:09.576703 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:09.576669 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:50:09.594572 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:09.594543 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:50:09.938238 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:09.938198 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" event={"ID":"12c4a382-3942-493e-84a4-85d061fc87e4","Type":"ContainerStarted","Data":"c54899381151cd0ab8dc487a6e05acfca6d49bbb0c8dec3bf6ab241c11b6ece4"} Nov 03 08:50:09.950135 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:09.950108 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qrhwd" Nov 03 08:50:10.058516 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:10.058452 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" podStartSLOduration=1.4448756170000001 podStartE2EDuration="2.058439022s" totalImagesPullingTime="613.563405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:10.056010729 +0000 UTC m=+642.980952967" watchObservedRunningTime="2026-11-03 08:50:10.058439022 +0000 UTC m=+642.983381241" Nov 03 08:50:10.439296 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:10.439251 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:16.903168 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:50:16.903098 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:50:17.337742 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.337694 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s"] Nov 03 08:50:17.341630 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.341604 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.401808 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.401772 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s"] Nov 03 08:50:17.447923 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.447880 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 08:50:17.476560 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.476517 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.476560 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.476562 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4537f2eb-9880-4ba9-a64a-0c25e21521dd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.476824 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.476634 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ftb\" (UniqueName: \"kubernetes.io/projected/4537f2eb-9880-4ba9-a64a-0c25e21521dd-kube-api-access-t7ftb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.476824 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.476722 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.476824 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.476790 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.477099 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.476830 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.577530 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.577467 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.577530 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.577532 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.577798 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.577603 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.577798 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.577628 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4537f2eb-9880-4ba9-a64a-0c25e21521dd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.577798 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.577669 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ftb\" (UniqueName: \"kubernetes.io/projected/4537f2eb-9880-4ba9-a64a-0c25e21521dd-kube-api-access-t7ftb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.577798 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.577695 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.578008 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.577964 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.578255 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.578226 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.578388 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.578289 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.580198 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.580176 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4537f2eb-9880-4ba9-a64a-0c25e21521dd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.580357 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.580339 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4537f2eb-9880-4ba9-a64a-0c25e21521dd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.638357 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.638282 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ftb\" (UniqueName: \"kubernetes.io/projected/4537f2eb-9880-4ba9-a64a-0c25e21521dd-kube-api-access-t7ftb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9884s\" (UID: \"4537f2eb-9880-4ba9-a64a-0c25e21521dd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:17.659844 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:17.659813 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:18.099785 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:18.099760 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s"] Nov 03 08:50:18.101726 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:50:18.101692 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4537f2eb_9880_4ba9_a64a_0c25e21521dd.slice/crio-2de00dd930704518da2b33af229fb23039d0fb5fb255d573f4e4d0cd1cc28d3c WatchSource:0}: Error finding container 2de00dd930704518da2b33af229fb23039d0fb5fb255d573f4e4d0cd1cc28d3c: Status 404 returned error can't find the container with id 2de00dd930704518da2b33af229fb23039d0fb5fb255d573f4e4d0cd1cc28d3c Nov 03 08:50:18.440005 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:18.439928 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:18.452688 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:18.452665 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:19.009342 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:19.009300 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" event={"ID":"4537f2eb-9880-4ba9-a64a-0c25e21521dd","Type":"ContainerStarted","Data":"0c2eb15e664dba8d9527e3686e734bd1a9b1187ed5b02c126d5419105a68b673"} Nov 03 08:50:19.009574 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:19.009356 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" event={"ID":"4537f2eb-9880-4ba9-a64a-0c25e21521dd","Type":"ContainerStarted","Data":"2de00dd930704518da2b33af229fb23039d0fb5fb255d573f4e4d0cd1cc28d3c"} Nov 03 08:50:19.020758 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:19.020737 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8" Nov 03 08:50:19.130285 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:19.130237 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" podStartSLOduration=1.5557914739999998 podStartE2EDuration="2.130223341s" totalImagesPullingTime="574.431867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:19.127965015 +0000 UTC m=+652.052907232" watchObservedRunningTime="2026-11-03 08:50:19.130223341 +0000 UTC m=+652.055165557" Nov 03 08:50:19.660796 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:19.660765 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:21.343025 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.342981 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6"] Nov 03 08:50:21.346823 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.346800 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.406660 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.406629 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6"] Nov 03 08:50:21.453079 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.453043 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 08:50:21.514750 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.514705 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.514929 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.514814 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9n2f\" (UniqueName: \"kubernetes.io/projected/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-kube-api-access-z9n2f\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.514929 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.514894 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.514929 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.514917 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.515127 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.514957 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.515127 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.515037 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.616306 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.616225 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.616306 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.616261 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.616306 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.616277 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.616633 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.616410 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.616633 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.616457 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.616633 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.616562 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9n2f\" (UniqueName: \"kubernetes.io/projected/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-kube-api-access-z9n2f\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.616825 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.616804 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.616886 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.616856 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.617076 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.617055 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.618911 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.618886 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.619337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.619316 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.677754 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.677724 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9n2f\" (UniqueName: \"kubernetes.io/projected/21ad0cfe-e3c6-4770-b0b3-7c512e7f1830-kube-api-access-z9n2f\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6\" (UID: \"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:21.958747 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:21.958646 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:22.348810 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:22.348780 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6"] Nov 03 08:50:22.350121 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:50:22.350094 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21ad0cfe_e3c6_4770_b0b3_7c512e7f1830.slice/crio-342617607ac8030b19ee3524219bfca5b711cad6f2bac95846c2b833e53bc0b5 WatchSource:0}: Error finding container 342617607ac8030b19ee3524219bfca5b711cad6f2bac95846c2b833e53bc0b5: Status 404 returned error can't find the container with id 342617607ac8030b19ee3524219bfca5b711cad6f2bac95846c2b833e53bc0b5 Nov 03 08:50:23.039099 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:23.039067 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" event={"ID":"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830","Type":"ContainerStarted","Data":"24a30fbe30abdf7b0e69030688cbf5d8ba757c0ea88dc66d718716825cbd32b4"} Nov 03 08:50:23.039231 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:23.039108 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" event={"ID":"21ad0cfe-e3c6-4770-b0b3-7c512e7f1830","Type":"ContainerStarted","Data":"342617607ac8030b19ee3524219bfca5b711cad6f2bac95846c2b833e53bc0b5"} Nov 03 08:50:23.184740 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:23.184681 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podStartSLOduration=1.581463802 podStartE2EDuration="2.184665381s" totalImagesPullingTime="603.201579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:23.183784948 +0000 UTC m=+656.108727165" watchObservedRunningTime="2026-11-03 08:50:23.184665381 +0000 UTC m=+656.109607598" Nov 03 08:50:23.959832 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:23.959789 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:27.660655 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:27.660609 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:27.677264 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:27.677237 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:28.078400 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:28.078368 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9884s" Nov 03 08:50:31.958872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:31.958835 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:31.972026 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:31.971996 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:32.104125 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.104095 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 08:50:32.612711 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.612674 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6"] Nov 03 08:50:32.616658 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.616640 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.629096 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.629064 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xzf\" (UniqueName: \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-kube-api-access-d4xzf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.629244 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.629117 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.629244 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.629198 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.629375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.629253 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.629375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.629340 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.629519 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.629502 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.629571 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.629551 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.629571 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.629566 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.629655 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.629612 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.730878 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.730847 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731044 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.730893 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731044 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.730912 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731044 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.730928 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731213 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.731051 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xzf\" (UniqueName: \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-kube-api-access-d4xzf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731213 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.731092 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731213 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.731117 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731213 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.731157 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731213 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.731187 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731441 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.731278 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731441 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.731352 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731547 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.731501 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731686 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.731668 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.731907 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.731888 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.733511 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.733487 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.733997 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.733978 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.791019 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.790988 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.791167 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.791096 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xzf\" (UniqueName: \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-kube-api-access-d4xzf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9n5v6\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:32.816840 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.816814 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6"] Nov 03 08:50:32.930038 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:32.929954 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:33.360920 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:33.360895 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6"] Nov 03 08:50:33.362375 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:50:33.362348 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b59d3b_6e05_4bd9_af71_7bf47cd6bc7f.slice/crio-c90848a0479113e90963d424a6a2ac4292cef255f180026fba40e6e6e5b3d6ca WatchSource:0}: Error finding container c90848a0479113e90963d424a6a2ac4292cef255f180026fba40e6e6e5b3d6ca: Status 404 returned error can't find the container with id c90848a0479113e90963d424a6a2ac4292cef255f180026fba40e6e6e5b3d6ca Nov 03 08:50:33.364400 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:33.364368 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:50:33.364517 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:33.364441 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:50:33.364517 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:33.364492 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:50:34.104740 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:34.104705 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" event={"ID":"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f","Type":"ContainerStarted","Data":"8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1"} Nov 03 08:50:34.104931 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:34.104753 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" event={"ID":"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f","Type":"ContainerStarted","Data":"c90848a0479113e90963d424a6a2ac4292cef255f180026fba40e6e6e5b3d6ca"} Nov 03 08:50:34.229771 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:34.229720 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" podStartSLOduration=2.22970432 podStartE2EDuration="2.22970432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:34.228278401 +0000 UTC m=+667.153220618" watchObservedRunningTime="2026-11-03 08:50:34.22970432 +0000 UTC m=+667.154646613" Nov 03 08:50:34.930861 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:34.930821 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:34.932436 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:34.932412 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 08:50:34.932527 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:34.932460 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" podUID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 08:50:35.931189 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:35.931150 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 08:50:35.931631 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:35.931205 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" podUID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 08:50:36.930884 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:36.930847 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 08:50:36.931031 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:36.930911 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" podUID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 08:50:37.930969 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:37.930930 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 08:50:37.931398 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:37.931041 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" podUID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 08:50:38.930987 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:38.930953 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 08:50:38.931373 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:38.931007 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" podUID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 08:50:39.931254 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:39.931217 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 08:50:39.931860 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:39.931294 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" podUID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 08:50:41.675909 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:41.675876 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:41.676382 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:41.676326 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:41.677157 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:41.677135 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:50:46.901540 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:50:46.901503 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:50:47.613127 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.613095 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s"] Nov 03 08:50:47.616898 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.616881 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.691753 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.691725 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.691753 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.691754 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.691936 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.691790 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn9g7\" (UniqueName: \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-kube-api-access-kn9g7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.691978 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.691936 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.691978 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.691964 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b243ef81-e040-408a-96e6-6ce5c7df25af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.692069 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.692051 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.692113 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.692097 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.692186 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.692173 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.692220 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.692199 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.792778 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.792740 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.792778 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.792780 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793016 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.792800 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793016 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.792834 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793016 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.792851 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793016 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.792891 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn9g7\" (UniqueName: \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-kube-api-access-kn9g7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793016 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.792926 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.793100 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b243ef81-e040-408a-96e6-6ce5c7df25af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.793164 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.793198 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.793245 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793604 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.793381 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793604 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.793566 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.793894 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.793873 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b243ef81-e040-408a-96e6-6ce5c7df25af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.795341 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.795314 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.795492 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.795459 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.845941 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.845905 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s"] Nov 03 08:50:47.852323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.852290 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.852466 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.852400 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn9g7\" (UniqueName: \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-kube-api-access-kn9g7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5gs6s\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:47.930461 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:47.930379 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:48.006019 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.005138 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dfcc7c588-7hvqz"] Nov 03 08:50:48.009294 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.009272 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dfcc7c588-7hvqz" Nov 03 08:50:48.063622 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.063588 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dfcc7c588-7hvqz"] Nov 03 08:50:48.096686 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.096656 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8ml\" (UniqueName: \"kubernetes.io/projected/ae051d13-de71-4752-b918-8bb60b21002d-kube-api-access-jq8ml\") pod \"authorino-6dfcc7c588-7hvqz\" (UID: \"ae051d13-de71-4752-b918-8bb60b21002d\") " pod="kuadrant-system/authorino-6dfcc7c588-7hvqz" Nov 03 08:50:48.096851 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.096701 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae051d13-de71-4752-b918-8bb60b21002d-tls-cert\") pod \"authorino-6dfcc7c588-7hvqz\" (UID: \"ae051d13-de71-4752-b918-8bb60b21002d\") " pod="kuadrant-system/authorino-6dfcc7c588-7hvqz" Nov 03 08:50:48.197819 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.197745 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8ml\" (UniqueName: \"kubernetes.io/projected/ae051d13-de71-4752-b918-8bb60b21002d-kube-api-access-jq8ml\") pod \"authorino-6dfcc7c588-7hvqz\" (UID: \"ae051d13-de71-4752-b918-8bb60b21002d\") " pod="kuadrant-system/authorino-6dfcc7c588-7hvqz" Nov 03 08:50:48.197819 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.197806 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae051d13-de71-4752-b918-8bb60b21002d-tls-cert\") pod \"authorino-6dfcc7c588-7hvqz\" (UID: \"ae051d13-de71-4752-b918-8bb60b21002d\") " pod="kuadrant-system/authorino-6dfcc7c588-7hvqz" Nov 03 08:50:48.200503 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.200461 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae051d13-de71-4752-b918-8bb60b21002d-tls-cert\") pod \"authorino-6dfcc7c588-7hvqz\" (UID: \"ae051d13-de71-4752-b918-8bb60b21002d\") " pod="kuadrant-system/authorino-6dfcc7c588-7hvqz" Nov 03 08:50:48.261883 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.261843 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8ml\" (UniqueName: \"kubernetes.io/projected/ae051d13-de71-4752-b918-8bb60b21002d-kube-api-access-jq8ml\") pod \"authorino-6dfcc7c588-7hvqz\" (UID: \"ae051d13-de71-4752-b918-8bb60b21002d\") " pod="kuadrant-system/authorino-6dfcc7c588-7hvqz" Nov 03 08:50:48.324547 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.324513 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dfcc7c588-7hvqz" Nov 03 08:50:48.361812 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.361780 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s"] Nov 03 08:50:48.364774 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.364738 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:50:48.364859 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.364818 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:50:48.364896 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.364860 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:50:48.704001 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:48.703975 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dfcc7c588-7hvqz"] Nov 03 08:50:48.706501 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:50:48.706450 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae051d13_de71_4752_b918_8bb60b21002d.slice/crio-a1cded9993d937b92653f23357c96e33b674f473389d3cb82535f8eb986ac96f WatchSource:0}: Error finding container a1cded9993d937b92653f23357c96e33b674f473389d3cb82535f8eb986ac96f: Status 404 returned error can't find the container with id a1cded9993d937b92653f23357c96e33b674f473389d3cb82535f8eb986ac96f Nov 03 08:50:49.203297 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:49.203264 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dfcc7c588-7hvqz" event={"ID":"ae051d13-de71-4752-b918-8bb60b21002d","Type":"ContainerStarted","Data":"a1cded9993d937b92653f23357c96e33b674f473389d3cb82535f8eb986ac96f"} Nov 03 08:50:49.203297 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:49.203300 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" event={"ID":"b243ef81-e040-408a-96e6-6ce5c7df25af","Type":"ContainerStarted","Data":"5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe"} Nov 03 08:50:49.203769 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:49.203327 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" event={"ID":"b243ef81-e040-408a-96e6-6ce5c7df25af","Type":"ContainerStarted","Data":"3226f3068f9e609b8e2c9d1fa244fffd326c70ee369dbe89815460ca69c96598"} Nov 03 08:50:49.327678 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:49.327627 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" podStartSLOduration=2.327611907 podStartE2EDuration="2.327611907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:49.32494409 +0000 UTC m=+682.249886306" watchObservedRunningTime="2026-11-03 08:50:49.327611907 +0000 UTC m=+682.252554117" Nov 03 08:50:49.931173 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:49.931143 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:49.932703 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:49.932673 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 08:50:49.932832 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:49.932717 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 08:50:50.931123 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:50.931088 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 08:50:50.931536 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:50.931140 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 08:50:51.215688 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.215600 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dfcc7c588-7hvqz" event={"ID":"ae051d13-de71-4752-b918-8bb60b21002d","Type":"ContainerStarted","Data":"8c9d8354f15ca875f73ddc2623dda569f814259b2b39fdbf6b1394e2b52891f9"} Nov 03 08:50:51.344524 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.344444 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6dfcc7c588-7hvqz" podStartSLOduration=2.764191033 podStartE2EDuration="4.344424401s" totalImagesPullingTime="1.580233368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:51.343993118 +0000 UTC m=+684.268935335" watchObservedRunningTime="2026-11-03 08:50:51.344424401 +0000 UTC m=+684.269366619" Nov 03 08:50:51.390999 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.390962 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cfccc9779-wgvs5"] Nov 03 08:50:51.391458 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.391423 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-cfccc9779-wgvs5" podUID="96182b4d-0fb2-4819-ba44-b4a3e069548d" containerName="authorino" containerID="cri-o://9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79" gracePeriod=30 Nov 03 08:50:51.787885 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.787860 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cfccc9779-wgvs5" Nov 03 08:50:51.887988 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.887948 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96182b4d-0fb2-4819-ba44-b4a3e069548d-kube-api-access-cdpqh\" (UniqueName: \"kubernetes.io/projected/96182b4d-0fb2-4819-ba44-b4a3e069548d-kube-api-access-cdpqh\") pod \"96182b4d-0fb2-4819-ba44-b4a3e069548d\" (UID: \"96182b4d-0fb2-4819-ba44-b4a3e069548d\") " Nov 03 08:50:51.888196 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.888069 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96182b4d-0fb2-4819-ba44-b4a3e069548d-tls-cert\" (UniqueName: \"kubernetes.io/secret/96182b4d-0fb2-4819-ba44-b4a3e069548d-tls-cert\") pod \"96182b4d-0fb2-4819-ba44-b4a3e069548d\" (UID: \"96182b4d-0fb2-4819-ba44-b4a3e069548d\") " Nov 03 08:50:51.891010 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.890973 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96182b4d-0fb2-4819-ba44-b4a3e069548d-kube-api-access-cdpqh" pod "96182b4d-0fb2-4819-ba44-b4a3e069548d" (UID: "96182b4d-0fb2-4819-ba44-b4a3e069548d"). InnerVolumeSpecName "kube-api-access-cdpqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:50:51.901149 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.901111 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96182b4d-0fb2-4819-ba44-b4a3e069548d-tls-cert" pod "96182b4d-0fb2-4819-ba44-b4a3e069548d" (UID: "96182b4d-0fb2-4819-ba44-b4a3e069548d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:50:51.930952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.930920 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 08:50:51.931110 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.930983 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 08:50:51.989334 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.989296 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdpqh\" (UniqueName: \"kubernetes.io/projected/96182b4d-0fb2-4819-ba44-b4a3e069548d-kube-api-access-cdpqh\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:50:51.989334 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:51.989326 5015 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96182b4d-0fb2-4819-ba44-b4a3e069548d-tls-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:50:52.224855 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:52.224820 5015 generic.go:356] "Generic (PLEG): container finished" podID="96182b4d-0fb2-4819-ba44-b4a3e069548d" containerID="9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79" exitCode=0 Nov 03 08:50:52.225025 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:52.224862 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cfccc9779-wgvs5" Nov 03 08:50:52.225025 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:52.224908 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cfccc9779-wgvs5" event={"ID":"96182b4d-0fb2-4819-ba44-b4a3e069548d","Type":"ContainerDied","Data":"9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79"} Nov 03 08:50:52.225025 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:52.224957 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cfccc9779-wgvs5" event={"ID":"96182b4d-0fb2-4819-ba44-b4a3e069548d","Type":"ContainerDied","Data":"16a2f0a4df2be0c94e168edd73fa93581a3cb4bf10e3da6598339ba095916c86"} Nov 03 08:50:52.225025 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:52.224980 5015 scope.go:122] "RemoveContainer" containerID="9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79" Nov 03 08:50:52.235346 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:52.235331 5015 scope.go:122] "RemoveContainer" containerID="9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79" Nov 03 08:50:52.235629 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:50:52.235610 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79\": container with ID starting with 9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79 not found: ID does not exist" containerID="9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79" Nov 03 08:50:52.235688 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:52.235637 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79"} err="failed to get container status \"9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79\": rpc error: code = NotFound desc = could not find container \"9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79\": container with ID starting with 9b8a8da0dc58ec8e6fe0bb5818e6195e894707f9e55c2b52a5859c3926aabb79 not found: ID does not exist" Nov 03 08:50:52.409383 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:52.409347 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cfccc9779-wgvs5"] Nov 03 08:50:52.414600 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:52.414572 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-cfccc9779-wgvs5"] Nov 03 08:50:52.931687 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:52.931646 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 08:50:52.931871 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:52.931708 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 08:50:53.210312 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:53.210229 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96182b4d-0fb2-4819-ba44-b4a3e069548d" path="/var/lib/kubelet/pods/96182b4d-0fb2-4819-ba44-b4a3e069548d/volumes" Nov 03 08:50:53.931388 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:53.931356 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 08:50:53.931572 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:53.931406 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 08:50:54.931229 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:54.931188 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 08:50:54.931637 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:54.931243 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 08:50:55.931599 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:55.931560 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 08:50:55.932022 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:55.931621 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 08:50:57.018043 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:57.018006 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:57.018560 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:57.018310 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:50:57.019183 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:50:57.019164 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 08:51:16.906403 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:51:16.906363 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:51:46.901858 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:51:46.901817 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:51:52.176631 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.176583 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc"] Nov 03 08:51:52.177454 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.177319 5015 cpu_manager.go:414] "containerMap: removing container" podUID="96182b4d-0fb2-4819-ba44-b4a3e069548d" containerName="authorino" Nov 03 08:51:52.177454 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.177338 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96182b4d-0fb2-4819-ba44-b4a3e069548d" containerName="authorino" Nov 03 08:51:52.177661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.177573 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="96182b4d-0fb2-4819-ba44-b4a3e069548d" containerName="authorino" Nov 03 08:51:52.183492 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.183443 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.242963 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.242930 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b6ef5ff5-openshift-default-dockercfg-8hpzg\"" Nov 03 08:51:52.262395 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.262356 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-podinfo\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.262612 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.262400 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dn25\" (UniqueName: \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-kube-api-access-4dn25\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.262612 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.262531 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b48b991-cda9-4672-85aa-77e3c5056f59-istiod-ca-cert\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.262612 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.262582 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-credential-socket\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.262749 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.262631 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-data\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.262749 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.262735 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-envoy\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.262893 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.262872 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-token\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.262941 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.262923 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-certs\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.262986 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.262972 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-socket\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.294899 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.294860 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc"] Nov 03 08:51:52.364429 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.364381 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-podinfo\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.364429 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.364422 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dn25\" (UniqueName: \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-kube-api-access-4dn25\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.364745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.364456 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b48b991-cda9-4672-85aa-77e3c5056f59-istiod-ca-cert\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.364745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.364524 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-credential-socket\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.364745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.364574 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-data\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.364745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.364600 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-envoy\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.364745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.364673 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-token\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.364745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.364712 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-certs\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.365047 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.364753 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-socket\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.365154 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.365132 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-certs\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.365246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.365228 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-credential-socket\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.365302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.365259 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-data\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.365394 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.365372 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-socket\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.365576 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.365558 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b48b991-cda9-4672-85aa-77e3c5056f59-istiod-ca-cert\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.367407 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.367386 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-envoy\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.367560 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.367543 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-podinfo\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.428777 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.428699 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-token\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.428914 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.428836 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dn25\" (UniqueName: \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-kube-api-access-4dn25\") pod \"e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.503425 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.503329 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:52.957825 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.957795 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc"] Nov 03 08:51:52.958938 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:51:52.958910 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b48b991_cda9_4672_85aa_77e3c5056f59.slice/crio-d44d0373cf077b403ae2859ffd88837b101ffa1b1efa2b9e37881f0fa0a87ef5 WatchSource:0}: Error finding container d44d0373cf077b403ae2859ffd88837b101ffa1b1efa2b9e37881f0fa0a87ef5: Status 404 returned error can't find the container with id d44d0373cf077b403ae2859ffd88837b101ffa1b1efa2b9e37881f0fa0a87ef5 Nov 03 08:51:52.961135 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.961100 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:51:52.961213 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.961166 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:51:52.961213 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:52.961197 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:51:53.605384 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:53.605347 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" event={"ID":"5b48b991-cda9-4672-85aa-77e3c5056f59","Type":"ContainerStarted","Data":"56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd"} Nov 03 08:51:53.605874 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:53.605404 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" event={"ID":"5b48b991-cda9-4672-85aa-77e3c5056f59","Type":"ContainerStarted","Data":"d44d0373cf077b403ae2859ffd88837b101ffa1b1efa2b9e37881f0fa0a87ef5"} Nov 03 08:51:53.795888 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:53.795839 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" podStartSLOduration=1.7958252890000002 podStartE2EDuration="1.795825289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:51:53.793887794 +0000 UTC m=+746.718830010" watchObservedRunningTime="2026-11-03 08:51:53.795825289 +0000 UTC m=+746.720767505" Nov 03 08:51:54.505033 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:54.504999 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:54.510141 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:54.510115 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:54.611829 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:54.611794 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:54.612820 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:54.612797 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:51:59.987969 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:59.987926 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb"] Nov 03 08:51:59.991780 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:51:59.991761 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:52:00.044467 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.044433 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkxvc\" (UniqueName: \"kubernetes.io/projected/38635fab-033d-4101-8467-b893aa1024e7-kube-api-access-gkxvc\") pod \"payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb\" (UID: \"38635fab-033d-4101-8467-b893aa1024e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:52:00.044647 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.044575 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38635fab-033d-4101-8467-b893aa1024e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb\" (UID: \"38635fab-033d-4101-8467-b893aa1024e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:52:00.050306 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.050277 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb"] Nov 03 08:52:00.052676 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.052652 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b6ef5ff5\"" Nov 03 08:52:00.138185 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.138147 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf"] Nov 03 08:52:00.141853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.141831 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:52:00.142992 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.142966 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b6ef5ff5-dockercfg-82dss\"" Nov 03 08:52:00.146469 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.146443 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38635fab-033d-4101-8467-b893aa1024e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb\" (UID: \"38635fab-033d-4101-8467-b893aa1024e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:52:00.157556 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.145463 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38635fab-033d-4101-8467-b893aa1024e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb\" (UID: \"38635fab-033d-4101-8467-b893aa1024e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:52:00.157786 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.157737 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkxvc\" (UniqueName: \"kubernetes.io/projected/38635fab-033d-4101-8467-b893aa1024e7-kube-api-access-gkxvc\") pod \"payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb\" (UID: \"38635fab-033d-4101-8467-b893aa1024e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:52:00.201827 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.201797 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf"] Nov 03 08:52:00.217690 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.217661 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkxvc\" (UniqueName: \"kubernetes.io/projected/38635fab-033d-4101-8467-b893aa1024e7-kube-api-access-gkxvc\") pod \"payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb\" (UID: \"38635fab-033d-4101-8467-b893aa1024e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:52:00.258738 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.258650 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr6tr\" (UniqueName: \"kubernetes.io/projected/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-kube-api-access-wr6tr\") pod \"payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf\" (UID: \"d1d7f676-6df9-47e6-a375-12b0e81bfdc6\") " pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:52:00.258879 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.258834 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-plugins-config-volume\") pod \"payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf\" (UID: \"d1d7f676-6df9-47e6-a375-12b0e81bfdc6\") " pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:52:00.283618 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.283575 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l"] Nov 03 08:52:00.287344 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.287324 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.333998 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.333959 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:52:00.359532 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.354218 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l"] Nov 03 08:52:00.360352 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.360290 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-data\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.360352 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.360368 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-credential-socket\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.360593 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.360414 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-socket\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.360593 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.360452 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-plugins-config-volume\") pod \"payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf\" (UID: \"d1d7f676-6df9-47e6-a375-12b0e81bfdc6\") " pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:52:00.360593 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.360469 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istiod-ca-cert\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.360593 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.360526 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-podinfo\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.360593 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.360580 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-token\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.361719 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.360602 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6tr\" (UniqueName: \"kubernetes.io/projected/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-kube-api-access-wr6tr\") pod \"payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf\" (UID: \"d1d7f676-6df9-47e6-a375-12b0e81bfdc6\") " pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:52:00.361719 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.360620 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-certs\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.361719 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.360696 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkpff\" (UniqueName: \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-kube-api-access-lkpff\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.361719 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.360727 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-envoy\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.361719 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.361559 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-plugins-config-volume\") pod \"payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf\" (UID: \"d1d7f676-6df9-47e6-a375-12b0e81bfdc6\") " pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:52:00.370705 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.370681 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7e658462-openshift-default-dockercfg-gznpk\"" Nov 03 08:52:00.425660 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.425592 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6tr\" (UniqueName: \"kubernetes.io/projected/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-kube-api-access-wr6tr\") pod \"payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf\" (UID: \"d1d7f676-6df9-47e6-a375-12b0e81bfdc6\") " pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:52:00.459273 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.459228 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:52:00.461685 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.461645 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkpff\" (UniqueName: \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-kube-api-access-lkpff\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.461854 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.461709 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-envoy\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.461854 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.461742 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-data\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.461854 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.461787 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-credential-socket\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.461854 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.461832 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-socket\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.462071 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.461867 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istiod-ca-cert\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.462071 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.461898 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-podinfo\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.462071 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.461965 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-token\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.462071 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.462003 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-certs\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.462274 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.462215 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-credential-socket\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.462666 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.462635 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istiod-ca-cert\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.462872 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.462848 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-socket\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.463290 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.463245 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-certs\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.463883 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.463862 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-data\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.464690 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.464668 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-envoy\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.465306 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.465280 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-podinfo\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.530200 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.530115 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkpff\" (UniqueName: \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-kube-api-access-lkpff\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.539585 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.532594 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-token\") pod \"e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.601049 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.601013 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:00.733110 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.733079 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb"] Nov 03 08:52:00.734812 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:52:00.734787 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38635fab_033d_4101_8467_b893aa1024e7.slice/crio-2f7bcef2305fa3657c40f5988d26f7344767315b154601fd2b782be016406663 WatchSource:0}: Error finding container 2f7bcef2305fa3657c40f5988d26f7344767315b154601fd2b782be016406663: Status 404 returned error can't find the container with id 2f7bcef2305fa3657c40f5988d26f7344767315b154601fd2b782be016406663 Nov 03 08:52:00.845693 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.845664 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf"] Nov 03 08:52:00.846825 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:52:00.846796 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d7f676_6df9_47e6_a375_12b0e81bfdc6.slice/crio-d51a662b4c85b36d45497d258d61e4446a342a569b7a0eadbce5b8b3b5c5fd83 WatchSource:0}: Error finding container d51a662b4c85b36d45497d258d61e4446a342a569b7a0eadbce5b8b3b5c5fd83: Status 404 returned error can't find the container with id d51a662b4c85b36d45497d258d61e4446a342a569b7a0eadbce5b8b3b5c5fd83 Nov 03 08:52:00.986607 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.986576 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l"] Nov 03 08:52:00.989040 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:52:00.989011 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081a7f81_c5fb_45f6_964e_5e2dc4e8fb96.slice/crio-137b146f321f282dc89b30d42843b0a1a353df033956c4b7b7a646dcec3c8eac WatchSource:0}: Error finding container 137b146f321f282dc89b30d42843b0a1a353df033956c4b7b7a646dcec3c8eac: Status 404 returned error can't find the container with id 137b146f321f282dc89b30d42843b0a1a353df033956c4b7b7a646dcec3c8eac Nov 03 08:52:00.991600 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.991568 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:52:00.991674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.991636 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:52:00.991674 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:00.991667 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:52:01.654240 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:01.654203 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" event={"ID":"d1d7f676-6df9-47e6-a375-12b0e81bfdc6","Type":"ContainerStarted","Data":"d51a662b4c85b36d45497d258d61e4446a342a569b7a0eadbce5b8b3b5c5fd83"} Nov 03 08:52:01.655769 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:01.655735 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" event={"ID":"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96","Type":"ContainerStarted","Data":"fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816"} Nov 03 08:52:01.655923 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:01.655788 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" event={"ID":"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96","Type":"ContainerStarted","Data":"137b146f321f282dc89b30d42843b0a1a353df033956c4b7b7a646dcec3c8eac"} Nov 03 08:52:01.656968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:01.656945 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" event={"ID":"38635fab-033d-4101-8467-b893aa1024e7","Type":"ContainerStarted","Data":"2f7bcef2305fa3657c40f5988d26f7344767315b154601fd2b782be016406663"} Nov 03 08:52:02.602381 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:02.602347 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:02.619511 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:02.608533 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:02.665397 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:02.665366 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:02.666667 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:02.666638 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:02.750930 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:02.750856 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" podStartSLOduration=2.750832313 podStartE2EDuration="2.750832313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:01.78872161 +0000 UTC m=+754.713663827" watchObservedRunningTime="2026-11-03 08:52:02.750832313 +0000 UTC m=+755.675774527" Nov 03 08:52:04.686509 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:04.685010 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" event={"ID":"38635fab-033d-4101-8467-b893aa1024e7","Type":"ContainerStarted","Data":"94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9"} Nov 03 08:52:04.690526 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:04.686733 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:52:04.692506 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:04.692183 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" event={"ID":"d1d7f676-6df9-47e6-a375-12b0e81bfdc6","Type":"ContainerStarted","Data":"b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b"} Nov 03 08:52:04.692506 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:04.692255 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:52:04.806327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:04.806269 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" podStartSLOduration=2.638114943 podStartE2EDuration="5.806252538s" totalImagesPullingTime="3.168137595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:04.804723929 +0000 UTC m=+757.729666148" watchObservedRunningTime="2026-11-03 08:52:04.806252538 +0000 UTC m=+757.731194786" Nov 03 08:52:04.923261 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:04.923202 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" podStartSLOduration=1.902865474 podStartE2EDuration="4.923187762s" totalImagesPullingTime="3.020322288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:04.921902504 +0000 UTC m=+757.846844739" watchObservedRunningTime="2026-11-03 08:52:04.923187762 +0000 UTC m=+757.848130011" Nov 03 08:52:05.012353 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.012316 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg"] Nov 03 08:52:05.016187 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.016168 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:52:05.071977 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.071945 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-82eec5e9\"" Nov 03 08:52:05.085623 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.072399 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-82eec5e9-dockercfg-29qql\"" Nov 03 08:52:05.093158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.093125 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg"] Nov 03 08:52:05.122406 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.122352 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83011dd2-5a6e-4535-9dbf-a78b5a807f19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg\" (UID: \"83011dd2-5a6e-4535-9dbf-a78b5a807f19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:52:05.122653 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.122436 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kj87\" (UniqueName: \"kubernetes.io/projected/83011dd2-5a6e-4535-9dbf-a78b5a807f19-kube-api-access-7kj87\") pod \"payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg\" (UID: \"83011dd2-5a6e-4535-9dbf-a78b5a807f19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:52:05.150721 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.150682 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x"] Nov 03 08:52:05.154805 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.154728 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:52:05.214968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.214940 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x"] Nov 03 08:52:05.223732 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.223701 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e84e39-42e3-4475-ad5e-8ec7d2438f60-plugins-config-volume\") pod \"payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x\" (UID: \"24e84e39-42e3-4475-ad5e-8ec7d2438f60\") " pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:52:05.223881 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.223739 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdcdq\" (UniqueName: \"kubernetes.io/projected/24e84e39-42e3-4475-ad5e-8ec7d2438f60-kube-api-access-wdcdq\") pod \"payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x\" (UID: \"24e84e39-42e3-4475-ad5e-8ec7d2438f60\") " pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:52:05.223881 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.223868 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83011dd2-5a6e-4535-9dbf-a78b5a807f19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg\" (UID: \"83011dd2-5a6e-4535-9dbf-a78b5a807f19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:52:05.224006 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.223914 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kj87\" (UniqueName: \"kubernetes.io/projected/83011dd2-5a6e-4535-9dbf-a78b5a807f19-kube-api-access-7kj87\") pod \"payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg\" (UID: \"83011dd2-5a6e-4535-9dbf-a78b5a807f19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:52:05.224649 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.224624 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83011dd2-5a6e-4535-9dbf-a78b5a807f19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg\" (UID: \"83011dd2-5a6e-4535-9dbf-a78b5a807f19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:52:05.284603 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.284522 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kj87\" (UniqueName: \"kubernetes.io/projected/83011dd2-5a6e-4535-9dbf-a78b5a807f19-kube-api-access-7kj87\") pod \"payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg\" (UID: \"83011dd2-5a6e-4535-9dbf-a78b5a807f19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:52:05.325567 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.325513 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e84e39-42e3-4475-ad5e-8ec7d2438f60-plugins-config-volume\") pod \"payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x\" (UID: \"24e84e39-42e3-4475-ad5e-8ec7d2438f60\") " pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:52:05.325567 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.325556 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdcdq\" (UniqueName: \"kubernetes.io/projected/24e84e39-42e3-4475-ad5e-8ec7d2438f60-kube-api-access-wdcdq\") pod \"payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x\" (UID: \"24e84e39-42e3-4475-ad5e-8ec7d2438f60\") " pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:52:05.326281 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.326258 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e84e39-42e3-4475-ad5e-8ec7d2438f60-plugins-config-volume\") pod \"payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x\" (UID: \"24e84e39-42e3-4475-ad5e-8ec7d2438f60\") " pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:52:05.331325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.331299 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:52:05.387778 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.387753 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdcdq\" (UniqueName: \"kubernetes.io/projected/24e84e39-42e3-4475-ad5e-8ec7d2438f60-kube-api-access-wdcdq\") pod \"payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x\" (UID: \"24e84e39-42e3-4475-ad5e-8ec7d2438f60\") " pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:52:05.467593 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.467563 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:52:05.716658 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.716625 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg"] Nov 03 08:52:05.717463 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:52:05.717436 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83011dd2_5a6e_4535_9dbf_a78b5a807f19.slice/crio-cdf4dd4049655ad01f3262cd29703adb5515e18cb1f7f02b5350bf58c7cc13e1 WatchSource:0}: Error finding container cdf4dd4049655ad01f3262cd29703adb5515e18cb1f7f02b5350bf58c7cc13e1: Status 404 returned error can't find the container with id cdf4dd4049655ad01f3262cd29703adb5515e18cb1f7f02b5350bf58c7cc13e1 Nov 03 08:52:05.852819 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:05.852783 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x"] Nov 03 08:52:05.853166 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:52:05.853138 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e84e39_42e3_4475_ad5e_8ec7d2438f60.slice/crio-e69c02191564f3031cd2e60f24c1b6270a6d978d118e7cb5e6c85961294e7a6a WatchSource:0}: Error finding container e69c02191564f3031cd2e60f24c1b6270a6d978d118e7cb5e6c85961294e7a6a: Status 404 returned error can't find the container with id e69c02191564f3031cd2e60f24c1b6270a6d978d118e7cb5e6c85961294e7a6a Nov 03 08:52:06.704904 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:06.704869 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" event={"ID":"83011dd2-5a6e-4535-9dbf-a78b5a807f19","Type":"ContainerStarted","Data":"c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665"} Nov 03 08:52:06.705106 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:06.704929 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" event={"ID":"83011dd2-5a6e-4535-9dbf-a78b5a807f19","Type":"ContainerStarted","Data":"cdf4dd4049655ad01f3262cd29703adb5515e18cb1f7f02b5350bf58c7cc13e1"} Nov 03 08:52:06.705106 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:06.705086 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:52:06.706622 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:06.706592 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" event={"ID":"24e84e39-42e3-4475-ad5e-8ec7d2438f60","Type":"ContainerStarted","Data":"7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8"} Nov 03 08:52:06.706755 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:06.706637 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" event={"ID":"24e84e39-42e3-4475-ad5e-8ec7d2438f60","Type":"ContainerStarted","Data":"e69c02191564f3031cd2e60f24c1b6270a6d978d118e7cb5e6c85961294e7a6a"} Nov 03 08:52:06.706755 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:06.706710 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:52:06.820815 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:06.820758 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" podStartSLOduration=2.820745344 podStartE2EDuration="2.820745344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:06.820102836 +0000 UTC m=+759.745045054" watchObservedRunningTime="2026-11-03 08:52:06.820745344 +0000 UTC m=+759.745687564" Nov 03 08:52:06.947521 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:06.939651 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" podStartSLOduration=1.9396296720000001 podStartE2EDuration="1.939629672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:06.938040187 +0000 UTC m=+759.862982425" watchObservedRunningTime="2026-11-03 08:52:06.939629672 +0000 UTC m=+759.864571893" Nov 03 08:52:07.561757 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.561721 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5"] Nov 03 08:52:07.565637 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.565619 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:52:07.620522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.620457 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6a1442a9-dockercfg-gzcbg\"" Nov 03 08:52:07.620522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.620507 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6a1442a9\"" Nov 03 08:52:07.628735 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.628706 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5"] Nov 03 08:52:07.654151 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.654098 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lq7l\" (UniqueName: \"kubernetes.io/projected/c121a127-8ef7-4c8b-8d9f-50d457f6f265-kube-api-access-6lq7l\") pod \"payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5\" (UID: \"c121a127-8ef7-4c8b-8d9f-50d457f6f265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:52:07.654545 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.654508 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c121a127-8ef7-4c8b-8d9f-50d457f6f265-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5\" (UID: \"c121a127-8ef7-4c8b-8d9f-50d457f6f265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:52:07.755521 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.755470 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c121a127-8ef7-4c8b-8d9f-50d457f6f265-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5\" (UID: \"c121a127-8ef7-4c8b-8d9f-50d457f6f265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:52:07.755683 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.755618 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lq7l\" (UniqueName: \"kubernetes.io/projected/c121a127-8ef7-4c8b-8d9f-50d457f6f265-kube-api-access-6lq7l\") pod \"payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5\" (UID: \"c121a127-8ef7-4c8b-8d9f-50d457f6f265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:52:07.756375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.756348 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c121a127-8ef7-4c8b-8d9f-50d457f6f265-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5\" (UID: \"c121a127-8ef7-4c8b-8d9f-50d457f6f265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:52:07.815122 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.815047 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lq7l\" (UniqueName: \"kubernetes.io/projected/c121a127-8ef7-4c8b-8d9f-50d457f6f265-kube-api-access-6lq7l\") pod \"payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5\" (UID: \"c121a127-8ef7-4c8b-8d9f-50d457f6f265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:52:07.883060 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:07.883032 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:52:08.261673 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:08.261627 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5"] Nov 03 08:52:08.266413 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:52:08.266375 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc121a127_8ef7_4c8b_8d9f_50d457f6f265.slice/crio-292b6cb47f148145d46f3df48ddc5cead7b97a0fae774fc48a01c09fcf8cc048 WatchSource:0}: Error finding container 292b6cb47f148145d46f3df48ddc5cead7b97a0fae774fc48a01c09fcf8cc048: Status 404 returned error can't find the container with id 292b6cb47f148145d46f3df48ddc5cead7b97a0fae774fc48a01c09fcf8cc048 Nov 03 08:52:08.719861 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:08.719760 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" event={"ID":"c121a127-8ef7-4c8b-8d9f-50d457f6f265","Type":"ContainerStarted","Data":"6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716"} Nov 03 08:52:08.719861 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:08.719824 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" event={"ID":"c121a127-8ef7-4c8b-8d9f-50d457f6f265","Type":"ContainerStarted","Data":"292b6cb47f148145d46f3df48ddc5cead7b97a0fae774fc48a01c09fcf8cc048"} Nov 03 08:52:08.719861 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:08.719851 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:52:08.845897 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:08.845832 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" podStartSLOduration=1.845817845 podStartE2EDuration="1.845817845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:08.842769638 +0000 UTC m=+761.767711892" watchObservedRunningTime="2026-11-03 08:52:08.845817845 +0000 UTC m=+761.770760062" Nov 03 08:52:12.545985 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.545946 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc"] Nov 03 08:52:12.549965 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.549942 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:52:12.607002 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.606956 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc"] Nov 03 08:52:12.645502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.644603 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jrf7\" (UniqueName: \"kubernetes.io/projected/669e5187-d582-4a12-bc17-604a1d8d7b18-kube-api-access-7jrf7\") pod \"payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc\" (UID: \"669e5187-d582-4a12-bc17-604a1d8d7b18\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:52:12.645502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.644729 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669e5187-d582-4a12-bc17-604a1d8d7b18-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc\" (UID: \"669e5187-d582-4a12-bc17-604a1d8d7b18\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:52:12.663909 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.663875 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-7e658462\"" Nov 03 08:52:12.663909 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.663888 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-7e658462-dockercfg-6kvr5\"" Nov 03 08:52:12.718730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.718695 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q"] Nov 03 08:52:12.723061 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.723040 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:52:12.746341 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.746154 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jrf7\" (UniqueName: \"kubernetes.io/projected/669e5187-d582-4a12-bc17-604a1d8d7b18-kube-api-access-7jrf7\") pod \"payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc\" (UID: \"669e5187-d582-4a12-bc17-604a1d8d7b18\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:52:12.746341 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.746310 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669e5187-d582-4a12-bc17-604a1d8d7b18-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc\" (UID: \"669e5187-d582-4a12-bc17-604a1d8d7b18\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:52:12.746856 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.746358 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf81cd02-012f-44af-a044-6fa92e98f887-plugins-config-volume\") pod \"payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q\" (UID: \"bf81cd02-012f-44af-a044-6fa92e98f887\") " pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:52:12.746856 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.746534 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kk6c\" (UniqueName: \"kubernetes.io/projected/bf81cd02-012f-44af-a044-6fa92e98f887-kube-api-access-5kk6c\") pod \"payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q\" (UID: \"bf81cd02-012f-44af-a044-6fa92e98f887\") " pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:52:12.747650 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.747619 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669e5187-d582-4a12-bc17-604a1d8d7b18-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc\" (UID: \"669e5187-d582-4a12-bc17-604a1d8d7b18\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:52:12.750943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.750919 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q"] Nov 03 08:52:12.805921 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.805842 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jrf7\" (UniqueName: \"kubernetes.io/projected/669e5187-d582-4a12-bc17-604a1d8d7b18-kube-api-access-7jrf7\") pod \"payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc\" (UID: \"669e5187-d582-4a12-bc17-604a1d8d7b18\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:52:12.849319 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.849285 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf81cd02-012f-44af-a044-6fa92e98f887-plugins-config-volume\") pod \"payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q\" (UID: \"bf81cd02-012f-44af-a044-6fa92e98f887\") " pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:52:12.849319 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.849317 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kk6c\" (UniqueName: \"kubernetes.io/projected/bf81cd02-012f-44af-a044-6fa92e98f887-kube-api-access-5kk6c\") pod \"payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q\" (UID: \"bf81cd02-012f-44af-a044-6fa92e98f887\") " pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:52:12.850091 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.850064 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf81cd02-012f-44af-a044-6fa92e98f887-plugins-config-volume\") pod \"payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q\" (UID: \"bf81cd02-012f-44af-a044-6fa92e98f887\") " pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:52:12.864133 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.864102 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:52:12.914621 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:12.914579 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kk6c\" (UniqueName: \"kubernetes.io/projected/bf81cd02-012f-44af-a044-6fa92e98f887-kube-api-access-5kk6c\") pod \"payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q\" (UID: \"bf81cd02-012f-44af-a044-6fa92e98f887\") " pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:52:13.036739 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:13.036709 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:52:13.245258 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:13.245227 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc"] Nov 03 08:52:13.246227 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:52:13.246205 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669e5187_d582_4a12_bc17_604a1d8d7b18.slice/crio-ce075831c0e57371577a123f63985dcd120299a76c7c2f5758a2e711bf13e6ae WatchSource:0}: Error finding container ce075831c0e57371577a123f63985dcd120299a76c7c2f5758a2e711bf13e6ae: Status 404 returned error can't find the container with id ce075831c0e57371577a123f63985dcd120299a76c7c2f5758a2e711bf13e6ae Nov 03 08:52:13.415898 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:13.415869 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q"] Nov 03 08:52:13.417390 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:52:13.417358 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf81cd02_012f_44af_a044_6fa92e98f887.slice/crio-eb661729ce3d69d8d7ba6f1f4fb8f9ac5ba3fc7892eba98fc5ac1065e0fe7120 WatchSource:0}: Error finding container eb661729ce3d69d8d7ba6f1f4fb8f9ac5ba3fc7892eba98fc5ac1065e0fe7120: Status 404 returned error can't find the container with id eb661729ce3d69d8d7ba6f1f4fb8f9ac5ba3fc7892eba98fc5ac1065e0fe7120 Nov 03 08:52:13.752670 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:13.752636 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" event={"ID":"bf81cd02-012f-44af-a044-6fa92e98f887","Type":"ContainerStarted","Data":"e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284"} Nov 03 08:52:13.753118 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:13.752688 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" event={"ID":"bf81cd02-012f-44af-a044-6fa92e98f887","Type":"ContainerStarted","Data":"eb661729ce3d69d8d7ba6f1f4fb8f9ac5ba3fc7892eba98fc5ac1065e0fe7120"} Nov 03 08:52:13.753118 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:13.752771 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:52:13.754455 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:13.754434 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" event={"ID":"669e5187-d582-4a12-bc17-604a1d8d7b18","Type":"ContainerStarted","Data":"5b24dd79a641c118ba6b565c057402af7abd440322d91c86c6d4f1bb3e84d342"} Nov 03 08:52:13.754607 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:13.754469 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" event={"ID":"669e5187-d582-4a12-bc17-604a1d8d7b18","Type":"ContainerStarted","Data":"ce075831c0e57371577a123f63985dcd120299a76c7c2f5758a2e711bf13e6ae"} Nov 03 08:52:13.754607 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:13.754528 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:52:13.869860 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:13.869807 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" podStartSLOduration=1.869792707 podStartE2EDuration="1.869792707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:13.868728364 +0000 UTC m=+766.793670600" watchObservedRunningTime="2026-11-03 08:52:13.869792707 +0000 UTC m=+766.794734947" Nov 03 08:52:13.986149 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:13.986084 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" podStartSLOduration=1.986065944 podStartE2EDuration="1.986065944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:13.985510175 +0000 UTC m=+766.910452414" watchObservedRunningTime="2026-11-03 08:52:13.986065944 +0000 UTC m=+766.911008161" Nov 03 08:52:15.702059 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:15.702030 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:52:16.709876 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:16.709847 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:52:16.903458 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:52:16.903424 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:52:17.715810 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:17.715772 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:52:17.716276 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:17.715912 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:52:19.729264 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:19.729231 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:52:23.901006 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:23.895562 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc"] Nov 03 08:52:23.908029 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:23.908001 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:23.967997 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:23.967960 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc"] Nov 03 08:52:23.987520 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:23.987456 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffbd\" (UniqueName: \"kubernetes.io/projected/0d7e59b6-313f-47bc-89ed-2e29da3ec030-kube-api-access-cffbd\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:23.987713 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:23.987569 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-maas-api-tls\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:23.987713 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:23.987599 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-metrics-certs\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:23.987713 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:23.987632 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7e59b6-313f-47bc-89ed-2e29da3ec030-openshift-service-ca\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:24.008327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.008297 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-7e658462-serving-cert\"" Nov 03 08:52:24.008511 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.008399 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f42g7\"" Nov 03 08:52:24.008511 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.008407 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 08:52:24.088934 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.088900 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-maas-api-tls\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:24.089125 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.088942 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-metrics-certs\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:24.089125 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.088993 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7e59b6-313f-47bc-89ed-2e29da3ec030-openshift-service-ca\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:24.089125 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:52:24.089043 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-serving-cert: secret "maas-api-e2e-derive-7e658462-serving-cert" not found Nov 03 08:52:24.089125 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:52:24.089117 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-maas-api-tls podName:0d7e59b6-313f-47bc-89ed-2e29da3ec030 nodeName:}" failed. No retries permitted until 2026-11-03 08:52:24.589097363 +0000 UTC m=+777.514039571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-maas-api-tls") pod "maas-api-e2e-derive-7e658462-594b5944cb-shbbc" (UID: "0d7e59b6-313f-47bc-89ed-2e29da3ec030") : secret "maas-api-e2e-derive-7e658462-serving-cert" not found Nov 03 08:52:24.089354 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.089159 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffbd\" (UniqueName: \"kubernetes.io/projected/0d7e59b6-313f-47bc-89ed-2e29da3ec030-kube-api-access-cffbd\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:24.089879 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.089854 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7e59b6-313f-47bc-89ed-2e29da3ec030-openshift-service-ca\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:24.092180 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.092155 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-metrics-certs\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:24.190401 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.189585 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffbd\" (UniqueName: \"kubernetes.io/projected/0d7e59b6-313f-47bc-89ed-2e29da3ec030-kube-api-access-cffbd\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:24.593512 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.593442 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-maas-api-tls\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:24.596199 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.596165 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-maas-api-tls\") pod \"maas-api-e2e-derive-7e658462-594b5944cb-shbbc\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:24.768777 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.768740 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:52:24.769804 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.769778 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:52:24.821686 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:24.821655 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:25.232976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:25.232947 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc"] Nov 03 08:52:25.233830 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:52:25.233805 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7e59b6_313f_47bc_89ed_2e29da3ec030.slice/crio-ca28c0da0c9b98bc4f517a707138d51a1621b86e578a916d66f8008209f80d4c WatchSource:0}: Error finding container ca28c0da0c9b98bc4f517a707138d51a1621b86e578a916d66f8008209f80d4c: Status 404 returned error can't find the container with id ca28c0da0c9b98bc4f517a707138d51a1621b86e578a916d66f8008209f80d4c Nov 03 08:52:25.842504 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:25.837622 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" event={"ID":"0d7e59b6-313f-47bc-89ed-2e29da3ec030","Type":"ContainerStarted","Data":"ca28c0da0c9b98bc4f517a707138d51a1621b86e578a916d66f8008209f80d4c"} Nov 03 08:52:29.896111 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:29.896069 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" event={"ID":"0d7e59b6-313f-47bc-89ed-2e29da3ec030","Type":"ContainerStarted","Data":"d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b"} Nov 03 08:52:29.896610 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:29.896200 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:30.045465 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:30.045384 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" podStartSLOduration=3.064858513 podStartE2EDuration="7.045364666s" totalImagesPullingTime="3.980506153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:30.041058463 +0000 UTC m=+782.966000681" watchObservedRunningTime="2026-11-03 08:52:30.045364666 +0000 UTC m=+782.970306883" Nov 03 08:52:32.924715 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:32.924624 5015 generic.go:356] "Generic (PLEG): container finished" podID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerID="56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd" exitCode=0 Nov 03 08:52:32.924715 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:32.924697 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" event={"ID":"5b48b991-cda9-4672-85aa-77e3c5056f59","Type":"ContainerDied","Data":"56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd"} Nov 03 08:52:32.925151 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:32.925076 5015 scope.go:122] "RemoveContainer" containerID="56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd" Nov 03 08:52:32.926444 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:32.926416 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:52:32.926522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:32.926507 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:52:32.926568 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:32.926539 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:52:33.933075 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:33.933034 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" event={"ID":"5b48b991-cda9-4672-85aa-77e3c5056f59","Type":"ContainerStarted","Data":"1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6"} Nov 03 08:52:34.507893 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:34.507861 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:52:34.508188 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:34.507905 5015 patch_prober.go:29] interesting pod/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:52:34.508325 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:34.508304 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 08:52:35.505836 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:35.505785 5015 patch_prober.go:29] interesting pod/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:52:35.506284 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:35.505856 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 08:52:35.912501 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:35.912390 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:52:36.505583 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:36.505537 5015 patch_prober.go:29] interesting pod/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:52:36.505862 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:36.505837 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 08:52:37.512297 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:37.511703 5015 patch_prober.go:29] interesting pod/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:52:37.512297 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:37.511824 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 08:52:38.506057 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:38.506023 5015 patch_prober.go:29] interesting pod/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:52:38.506235 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:38.506094 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 08:52:39.510343 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:39.510313 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:52:39.510770 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:39.510689 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:52:39.511337 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:39.511317 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:52:42.036262 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:42.036230 5015 generic.go:356] "Generic (PLEG): container finished" podID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerID="fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816" exitCode=0 Nov 03 08:52:42.036760 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:42.036309 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" event={"ID":"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96","Type":"ContainerDied","Data":"fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816"} Nov 03 08:52:42.036760 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:42.036721 5015 scope.go:122] "RemoveContainer" containerID="fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816" Nov 03 08:52:42.038033 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:42.038001 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:52:42.038159 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:42.038086 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:52:42.038159 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:42.038133 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:52:43.091955 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:43.091915 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" event={"ID":"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96","Type":"ContainerStarted","Data":"929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205"} Nov 03 08:52:43.602101 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:43.602062 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:43.603215 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:43.603183 5015 patch_prober.go:29] interesting pod/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 08:52:43.603338 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:43.603251 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 08:52:44.601633 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:44.601595 5015 patch_prober.go:29] interesting pod/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 08:52:44.602062 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:44.601660 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 08:52:45.601711 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:45.601676 5015 patch_prober.go:29] interesting pod/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 08:52:45.602097 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:45.601730 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 08:52:46.601924 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:46.601889 5015 patch_prober.go:29] interesting pod/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 08:52:46.602328 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:46.601941 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 08:52:46.902310 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:52:46.902229 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:52:47.601613 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:47.601584 5015 patch_prober.go:29] interesting pod/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 08:52:47.601792 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:47.601632 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 08:52:47.665421 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:47.665368 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:48.606175 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:48.606142 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:48.607288 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:48.607266 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:52:52.420940 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.420906 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b"] Nov 03 08:52:52.425059 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.425042 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:52:52.479831 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.479797 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 08:52:52.505445 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.505397 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:52:52.505677 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.505561 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp82t\" (UniqueName: \"kubernetes.io/projected/6d8ec563-49e9-4d9f-b595-f066173631b2-kube-api-access-rp82t\") pod \"maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:52:52.505677 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.505609 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:52:52.529436 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.529393 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 08:52:52.534831 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.534787 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b"] Nov 03 08:52:52.606433 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.606390 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp82t\" (UniqueName: \"kubernetes.io/projected/6d8ec563-49e9-4d9f-b595-f066173631b2-kube-api-access-rp82t\") pod \"maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:52:52.606625 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.606468 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:52:52.606625 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.606616 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:52:52.607244 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.607214 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:52:52.609336 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.609311 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:52:52.668405 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.668363 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp82t\" (UniqueName: \"kubernetes.io/projected/6d8ec563-49e9-4d9f-b595-f066173631b2-kube-api-access-rp82t\") pod \"maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:52:52.739272 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:52.739239 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:52:53.120782 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:53.120751 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b"] Nov 03 08:52:53.121650 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:52:53.121623 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8ec563_49e9_4d9f_b595_f066173631b2.slice/crio-11a781c32dc22dfd4a86c36d630456f407573d9f00eb06f518ff0b075ed70b4a WatchSource:0}: Error finding container 11a781c32dc22dfd4a86c36d630456f407573d9f00eb06f518ff0b075ed70b4a: Status 404 returned error can't find the container with id 11a781c32dc22dfd4a86c36d630456f407573d9f00eb06f518ff0b075ed70b4a Nov 03 08:52:53.153021 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:53.152992 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" event={"ID":"6d8ec563-49e9-4d9f-b595-f066173631b2","Type":"ContainerStarted","Data":"11a781c32dc22dfd4a86c36d630456f407573d9f00eb06f518ff0b075ed70b4a"} Nov 03 08:52:59.190418 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:59.190329 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5"] Nov 03 08:52:59.193939 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:59.193910 5015 generic.go:356] "Generic (PLEG): container finished" podID="6d8ec563-49e9-4d9f-b595-f066173631b2" containerID="241649e1f156b3d3653047ec4679a1d93280cfb2712413cdbf6c2558989f9951" exitCode=0 Nov 03 08:52:59.194068 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:52:59.193997 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" event={"ID":"6d8ec563-49e9-4d9f-b595-f066173631b2","Type":"ContainerDied","Data":"241649e1f156b3d3653047ec4679a1d93280cfb2712413cdbf6c2558989f9951"} Nov 03 08:53:00.391584 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:00.391556 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:53:00.494743 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:00.494710 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-xks-ca\") pod \"6d8ec563-49e9-4d9f-b595-f066173631b2\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " Nov 03 08:53:00.494929 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:00.494770 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8ec563-49e9-4d9f-b595-f066173631b2-kube-api-access-rp82t\" (UniqueName: \"kubernetes.io/projected/6d8ec563-49e9-4d9f-b595-f066173631b2-kube-api-access-rp82t\") pod \"6d8ec563-49e9-4d9f-b595-f066173631b2\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " Nov 03 08:53:00.494997 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:00.494941 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-openshift-ca\") pod \"6d8ec563-49e9-4d9f-b595-f066173631b2\" (UID: \"6d8ec563-49e9-4d9f-b595-f066173631b2\") " Nov 03 08:53:00.495358 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:00.495328 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-openshift-ca" pod "6d8ec563-49e9-4d9f-b595-f066173631b2" (UID: "6d8ec563-49e9-4d9f-b595-f066173631b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:00.497729 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:00.497696 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-xks-ca" pod "6d8ec563-49e9-4d9f-b595-f066173631b2" (UID: "6d8ec563-49e9-4d9f-b595-f066173631b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:53:00.497818 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:00.497700 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8ec563-49e9-4d9f-b595-f066173631b2-kube-api-access-rp82t" pod "6d8ec563-49e9-4d9f-b595-f066173631b2" (UID: "6d8ec563-49e9-4d9f-b595-f066173631b2"). InnerVolumeSpecName "kube-api-access-rp82t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:00.596970 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:00.596925 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:00.596970 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:00.596969 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8ec563-49e9-4d9f-b595-f066173631b2-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:00.597210 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:00.596982 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp82t\" (UniqueName: \"kubernetes.io/projected/6d8ec563-49e9-4d9f-b595-f066173631b2-kube-api-access-rp82t\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:01.204049 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:01.204020 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x"] Nov 03 08:53:01.208192 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:01.208170 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" Nov 03 08:53:01.208352 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:01.208204 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b" event={"ID":"6d8ec563-49e9-4d9f-b595-f066173631b2","Type":"ContainerDied","Data":"11a781c32dc22dfd4a86c36d630456f407573d9f00eb06f518ff0b075ed70b4a"} Nov 03 08:53:01.208352 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:01.208248 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11a781c32dc22dfd4a86c36d630456f407573d9f00eb06f518ff0b075ed70b4a" Nov 03 08:53:01.253244 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:01.253214 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg"] Nov 03 08:53:04.191393 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:04.191351 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" podUID="c121a127-8ef7-4c8b-8d9f-50d457f6f265" containerName="payload-pre-processing" containerID="cri-o://6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716" gracePeriod=25 Nov 03 08:53:04.213303 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:04.213278 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf"] Nov 03 08:53:04.325864 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:04.325829 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb"] Nov 03 08:53:04.453910 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:04.453854 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:53:04.540441 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:04.540410 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c121a127-8ef7-4c8b-8d9f-50d457f6f265-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c121a127-8ef7-4c8b-8d9f-50d457f6f265-plugins-config-volume\") pod \"c121a127-8ef7-4c8b-8d9f-50d457f6f265\" (UID: \"c121a127-8ef7-4c8b-8d9f-50d457f6f265\") " Nov 03 08:53:04.540635 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:04.540518 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c121a127-8ef7-4c8b-8d9f-50d457f6f265-kube-api-access-6lq7l\" (UniqueName: \"kubernetes.io/projected/c121a127-8ef7-4c8b-8d9f-50d457f6f265-kube-api-access-6lq7l\") pod \"c121a127-8ef7-4c8b-8d9f-50d457f6f265\" (UID: \"c121a127-8ef7-4c8b-8d9f-50d457f6f265\") " Nov 03 08:53:04.540797 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:04.540775 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c121a127-8ef7-4c8b-8d9f-50d457f6f265-plugins-config-volume" pod "c121a127-8ef7-4c8b-8d9f-50d457f6f265" (UID: "c121a127-8ef7-4c8b-8d9f-50d457f6f265"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:04.542881 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:04.542855 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c121a127-8ef7-4c8b-8d9f-50d457f6f265-kube-api-access-6lq7l" pod "c121a127-8ef7-4c8b-8d9f-50d457f6f265" (UID: "c121a127-8ef7-4c8b-8d9f-50d457f6f265"). InnerVolumeSpecName "kube-api-access-6lq7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:04.642512 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:04.642447 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lq7l\" (UniqueName: \"kubernetes.io/projected/c121a127-8ef7-4c8b-8d9f-50d457f6f265-kube-api-access-6lq7l\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:04.642512 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:04.642503 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c121a127-8ef7-4c8b-8d9f-50d457f6f265-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:05.233317 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:05.233283 5015 generic.go:356] "Generic (PLEG): container finished" podID="c121a127-8ef7-4c8b-8d9f-50d457f6f265" containerID="6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716" exitCode=0 Nov 03 08:53:05.233732 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:05.233327 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" Nov 03 08:53:05.233732 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:05.233366 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" event={"ID":"c121a127-8ef7-4c8b-8d9f-50d457f6f265","Type":"ContainerDied","Data":"6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716"} Nov 03 08:53:05.233732 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:05.233427 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5" event={"ID":"c121a127-8ef7-4c8b-8d9f-50d457f6f265","Type":"ContainerDied","Data":"292b6cb47f148145d46f3df48ddc5cead7b97a0fae774fc48a01c09fcf8cc048"} Nov 03 08:53:05.233732 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:05.233458 5015 scope.go:122] "RemoveContainer" containerID="6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716" Nov 03 08:53:05.243630 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:05.243611 5015 scope.go:122] "RemoveContainer" containerID="6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716" Nov 03 08:53:05.243935 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:05.243912 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716\": container with ID starting with 6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716 not found: ID does not exist" containerID="6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716" Nov 03 08:53:05.244019 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:05.243943 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716"} err="failed to get container status \"6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716\": rpc error: code = NotFound desc = could not find container \"6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716\": container with ID starting with 6613ecc946ea9096731e2d687b2547fe124b6a50dc86233003aa99503d7ef716 not found: ID does not exist" Nov 03 08:53:05.403105 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:05.403074 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5"] Nov 03 08:53:05.408433 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:05.408406 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a1442a9-7d76cdff6c-qjxn5"] Nov 03 08:53:06.204643 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.204584 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" podUID="24e84e39-42e3-4475-ad5e-8ec7d2438f60" containerName="payload-processing" containerID="cri-o://7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8" gracePeriod=25 Nov 03 08:53:06.253992 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.253959 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" podUID="83011dd2-5a6e-4535-9dbf-a78b5a807f19" containerName="payload-pre-processing" containerID="cri-o://c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665" gracePeriod=25 Nov 03 08:53:06.511356 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.511328 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:53:06.538736 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.538713 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:53:06.662814 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.662781 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83011dd2-5a6e-4535-9dbf-a78b5a807f19-kube-api-access-7kj87\" (UniqueName: \"kubernetes.io/projected/83011dd2-5a6e-4535-9dbf-a78b5a807f19-kube-api-access-7kj87\") pod \"83011dd2-5a6e-4535-9dbf-a78b5a807f19\" (UID: \"83011dd2-5a6e-4535-9dbf-a78b5a807f19\") " Nov 03 08:53:06.662975 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.662846 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83011dd2-5a6e-4535-9dbf-a78b5a807f19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83011dd2-5a6e-4535-9dbf-a78b5a807f19-plugins-config-volume\") pod \"83011dd2-5a6e-4535-9dbf-a78b5a807f19\" (UID: \"83011dd2-5a6e-4535-9dbf-a78b5a807f19\") " Nov 03 08:53:06.662975 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.662926 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e84e39-42e3-4475-ad5e-8ec7d2438f60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e84e39-42e3-4475-ad5e-8ec7d2438f60-plugins-config-volume\") pod \"24e84e39-42e3-4475-ad5e-8ec7d2438f60\" (UID: \"24e84e39-42e3-4475-ad5e-8ec7d2438f60\") " Nov 03 08:53:06.663086 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.663072 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e84e39-42e3-4475-ad5e-8ec7d2438f60-kube-api-access-wdcdq\" (UniqueName: \"kubernetes.io/projected/24e84e39-42e3-4475-ad5e-8ec7d2438f60-kube-api-access-wdcdq\") pod \"24e84e39-42e3-4475-ad5e-8ec7d2438f60\" (UID: \"24e84e39-42e3-4475-ad5e-8ec7d2438f60\") " Nov 03 08:53:06.663348 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.663317 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e84e39-42e3-4475-ad5e-8ec7d2438f60-plugins-config-volume" pod "24e84e39-42e3-4475-ad5e-8ec7d2438f60" (UID: "24e84e39-42e3-4475-ad5e-8ec7d2438f60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:06.663348 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.663317 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83011dd2-5a6e-4535-9dbf-a78b5a807f19-plugins-config-volume" pod "83011dd2-5a6e-4535-9dbf-a78b5a807f19" (UID: "83011dd2-5a6e-4535-9dbf-a78b5a807f19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:06.663605 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.663405 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83011dd2-5a6e-4535-9dbf-a78b5a807f19-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:06.663605 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.663425 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e84e39-42e3-4475-ad5e-8ec7d2438f60-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:06.665170 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.665140 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83011dd2-5a6e-4535-9dbf-a78b5a807f19-kube-api-access-7kj87" pod "83011dd2-5a6e-4535-9dbf-a78b5a807f19" (UID: "83011dd2-5a6e-4535-9dbf-a78b5a807f19"). InnerVolumeSpecName "kube-api-access-7kj87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:06.665397 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.665377 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e84e39-42e3-4475-ad5e-8ec7d2438f60-kube-api-access-wdcdq" pod "24e84e39-42e3-4475-ad5e-8ec7d2438f60" (UID: "24e84e39-42e3-4475-ad5e-8ec7d2438f60"). InnerVolumeSpecName "kube-api-access-wdcdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:06.764177 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.764151 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdcdq\" (UniqueName: \"kubernetes.io/projected/24e84e39-42e3-4475-ad5e-8ec7d2438f60-kube-api-access-wdcdq\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:06.764177 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:06.764177 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kj87\" (UniqueName: \"kubernetes.io/projected/83011dd2-5a6e-4535-9dbf-a78b5a807f19-kube-api-access-7kj87\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:07.203576 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.203463 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c121a127-8ef7-4c8b-8d9f-50d457f6f265" path="/var/lib/kubelet/pods/c121a127-8ef7-4c8b-8d9f-50d457f6f265/volumes" Nov 03 08:53:07.247745 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.247717 5015 generic.go:356] "Generic (PLEG): container finished" podID="83011dd2-5a6e-4535-9dbf-a78b5a807f19" containerID="c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665" exitCode=0 Nov 03 08:53:07.247857 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.247762 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" Nov 03 08:53:07.247857 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.247802 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" event={"ID":"83011dd2-5a6e-4535-9dbf-a78b5a807f19","Type":"ContainerDied","Data":"c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665"} Nov 03 08:53:07.247857 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.247848 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg" event={"ID":"83011dd2-5a6e-4535-9dbf-a78b5a807f19","Type":"ContainerDied","Data":"cdf4dd4049655ad01f3262cd29703adb5515e18cb1f7f02b5350bf58c7cc13e1"} Nov 03 08:53:07.247988 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.247874 5015 scope.go:122] "RemoveContainer" containerID="c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665" Nov 03 08:53:07.249248 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.249225 5015 generic.go:356] "Generic (PLEG): container finished" podID="24e84e39-42e3-4475-ad5e-8ec7d2438f60" containerID="7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8" exitCode=0 Nov 03 08:53:07.249351 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.249268 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" Nov 03 08:53:07.249351 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.249298 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" event={"ID":"24e84e39-42e3-4475-ad5e-8ec7d2438f60","Type":"ContainerDied","Data":"7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8"} Nov 03 08:53:07.249351 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.249335 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x" event={"ID":"24e84e39-42e3-4475-ad5e-8ec7d2438f60","Type":"ContainerDied","Data":"e69c02191564f3031cd2e60f24c1b6270a6d978d118e7cb5e6c85961294e7a6a"} Nov 03 08:53:07.258007 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.257840 5015 scope.go:122] "RemoveContainer" containerID="c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665" Nov 03 08:53:07.258268 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:07.258132 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665\": container with ID starting with c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665 not found: ID does not exist" containerID="c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665" Nov 03 08:53:07.258268 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.258156 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665"} err="failed to get container status \"c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665\": rpc error: code = NotFound desc = could not find container \"c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665\": container with ID starting with c4a9f9fedb1117d928523ae443b11cbe83e61f9db98e681bd6d726d78391d665 not found: ID does not exist" Nov 03 08:53:07.258268 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.258173 5015 scope.go:122] "RemoveContainer" containerID="7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8" Nov 03 08:53:07.267004 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.266989 5015 scope.go:122] "RemoveContainer" containerID="7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8" Nov 03 08:53:07.267282 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:07.267262 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8\": container with ID starting with 7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8 not found: ID does not exist" containerID="7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8" Nov 03 08:53:07.267341 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.267290 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8"} err="failed to get container status \"7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8\": rpc error: code = NotFound desc = could not find container \"7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8\": container with ID starting with 7a41e67f466ba835716ac9a288dff024da2c77ec86a9011bce5747848c3180a8 not found: ID does not exist" Nov 03 08:53:07.417804 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.417772 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x"] Nov 03 08:53:07.426791 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.426765 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82eec5e9-7b8944658c-ps68x"] Nov 03 08:53:07.540755 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.540725 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg"] Nov 03 08:53:07.547100 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:07.547075 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82eec5e9-6b78f87d55-28nlg"] Nov 03 08:53:09.204017 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.203987 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e84e39-42e3-4475-ad5e-8ec7d2438f60" path="/var/lib/kubelet/pods/24e84e39-42e3-4475-ad5e-8ec7d2438f60/volumes" Nov 03 08:53:09.204406 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.204335 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83011dd2-5a6e-4535-9dbf-a78b5a807f19" path="/var/lib/kubelet/pods/83011dd2-5a6e-4535-9dbf-a78b5a807f19/volumes" Nov 03 08:53:09.213944 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.213917 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" podUID="d1d7f676-6df9-47e6-a375-12b0e81bfdc6" containerName="payload-processing" containerID="cri-o://b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b" gracePeriod=25 Nov 03 08:53:09.326866 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.326816 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" podUID="38635fab-033d-4101-8467-b893aa1024e7" containerName="payload-pre-processing" containerID="cri-o://94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9" gracePeriod=25 Nov 03 08:53:09.491062 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.491030 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:53:09.500078 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.500045 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-kube-api-access-wr6tr\" (UniqueName: \"kubernetes.io/projected/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-kube-api-access-wr6tr\") pod \"d1d7f676-6df9-47e6-a375-12b0e81bfdc6\" (UID: \"d1d7f676-6df9-47e6-a375-12b0e81bfdc6\") " Nov 03 08:53:09.500235 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.500208 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-plugins-config-volume\") pod \"d1d7f676-6df9-47e6-a375-12b0e81bfdc6\" (UID: \"d1d7f676-6df9-47e6-a375-12b0e81bfdc6\") " Nov 03 08:53:09.500879 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.500850 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-plugins-config-volume" pod "d1d7f676-6df9-47e6-a375-12b0e81bfdc6" (UID: "d1d7f676-6df9-47e6-a375-12b0e81bfdc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:09.503101 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.503036 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-kube-api-access-wr6tr" pod "d1d7f676-6df9-47e6-a375-12b0e81bfdc6" (UID: "d1d7f676-6df9-47e6-a375-12b0e81bfdc6"). InnerVolumeSpecName "kube-api-access-wr6tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:09.579224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.579197 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:53:09.600918 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.600884 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38635fab-033d-4101-8467-b893aa1024e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38635fab-033d-4101-8467-b893aa1024e7-plugins-config-volume\") pod \"38635fab-033d-4101-8467-b893aa1024e7\" (UID: \"38635fab-033d-4101-8467-b893aa1024e7\") " Nov 03 08:53:09.601055 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.601022 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38635fab-033d-4101-8467-b893aa1024e7-kube-api-access-gkxvc\" (UniqueName: \"kubernetes.io/projected/38635fab-033d-4101-8467-b893aa1024e7-kube-api-access-gkxvc\") pod \"38635fab-033d-4101-8467-b893aa1024e7\" (UID: \"38635fab-033d-4101-8467-b893aa1024e7\") " Nov 03 08:53:09.601236 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.601213 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38635fab-033d-4101-8467-b893aa1024e7-plugins-config-volume" pod "38635fab-033d-4101-8467-b893aa1024e7" (UID: "38635fab-033d-4101-8467-b893aa1024e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:09.601341 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.601320 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:09.601379 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.601348 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr6tr\" (UniqueName: \"kubernetes.io/projected/d1d7f676-6df9-47e6-a375-12b0e81bfdc6-kube-api-access-wr6tr\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:09.601379 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.601363 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38635fab-033d-4101-8467-b893aa1024e7-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:09.603314 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.603295 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38635fab-033d-4101-8467-b893aa1024e7-kube-api-access-gkxvc" pod "38635fab-033d-4101-8467-b893aa1024e7" (UID: "38635fab-033d-4101-8467-b893aa1024e7"). InnerVolumeSpecName "kube-api-access-gkxvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:09.702705 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:09.702666 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkxvc\" (UniqueName: \"kubernetes.io/projected/38635fab-033d-4101-8467-b893aa1024e7-kube-api-access-gkxvc\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:10.274770 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.274737 5015 generic.go:356] "Generic (PLEG): container finished" podID="d1d7f676-6df9-47e6-a375-12b0e81bfdc6" containerID="b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b" exitCode=0 Nov 03 08:53:10.275256 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.274786 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" Nov 03 08:53:10.275256 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.274820 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" event={"ID":"d1d7f676-6df9-47e6-a375-12b0e81bfdc6","Type":"ContainerDied","Data":"b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b"} Nov 03 08:53:10.275256 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.274883 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf" event={"ID":"d1d7f676-6df9-47e6-a375-12b0e81bfdc6","Type":"ContainerDied","Data":"d51a662b4c85b36d45497d258d61e4446a342a569b7a0eadbce5b8b3b5c5fd83"} Nov 03 08:53:10.275256 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.274915 5015 scope.go:122] "RemoveContainer" containerID="b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b" Nov 03 08:53:10.276634 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.276579 5015 generic.go:356] "Generic (PLEG): container finished" podID="38635fab-033d-4101-8467-b893aa1024e7" containerID="94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9" exitCode=0 Nov 03 08:53:10.276634 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.276633 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" Nov 03 08:53:10.276834 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.276709 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" event={"ID":"38635fab-033d-4101-8467-b893aa1024e7","Type":"ContainerDied","Data":"94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9"} Nov 03 08:53:10.276834 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.276750 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb" event={"ID":"38635fab-033d-4101-8467-b893aa1024e7","Type":"ContainerDied","Data":"2f7bcef2305fa3657c40f5988d26f7344767315b154601fd2b782be016406663"} Nov 03 08:53:10.288611 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.288588 5015 scope.go:122] "RemoveContainer" containerID="b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b" Nov 03 08:53:10.291439 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:10.291405 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b\": container with ID starting with b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b not found: ID does not exist" containerID="b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b" Nov 03 08:53:10.291587 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.291447 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b"} err="failed to get container status \"b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b\": rpc error: code = NotFound desc = could not find container \"b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b\": container with ID starting with b2e290dd4b76b20355358303e62e9c1c887179102eacb098082827f20483f56b not found: ID does not exist" Nov 03 08:53:10.291587 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.291466 5015 scope.go:122] "RemoveContainer" containerID="94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9" Nov 03 08:53:10.302114 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.302093 5015 scope.go:122] "RemoveContainer" containerID="94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9" Nov 03 08:53:10.302432 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:10.302405 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9\": container with ID starting with 94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9 not found: ID does not exist" containerID="94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9" Nov 03 08:53:10.302561 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.302443 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9"} err="failed to get container status \"94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9\": rpc error: code = NotFound desc = could not find container \"94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9\": container with ID starting with 94982262feb52f9c9999e7c965d36016de4cef3926b1a175a9546b2a5d144eb9 not found: ID does not exist" Nov 03 08:53:10.446097 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.446063 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb"] Nov 03 08:53:10.496247 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.496170 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b6ef5ff5-6fc484d9f-25hwb"] Nov 03 08:53:10.566147 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.566103 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf"] Nov 03 08:53:10.574105 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:10.574077 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b6ef5ff5-8556786649-2wnjf"] Nov 03 08:53:11.204650 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:11.204620 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38635fab-033d-4101-8467-b893aa1024e7" path="/var/lib/kubelet/pods/38635fab-033d-4101-8467-b893aa1024e7/volumes" Nov 03 08:53:11.205031 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:11.205018 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1d7f676-6df9-47e6-a375-12b0e81bfdc6" path="/var/lib/kubelet/pods/d1d7f676-6df9-47e6-a375-12b0e81bfdc6/volumes" Nov 03 08:53:11.900914 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:11.900878 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b"] Nov 03 08:53:11.955914 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:11.955567 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6ef5ff5-9td5b"] Nov 03 08:53:11.977464 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:11.977429 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc"] Nov 03 08:53:11.977752 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:11.977721 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" containerID="cri-o://1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6" gracePeriod=30 Nov 03 08:53:12.401013 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.400973 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2"] Nov 03 08:53:12.401517 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401498 5015 cpu_manager.go:414] "containerMap: removing container" podUID="38635fab-033d-4101-8467-b893aa1024e7" containerName="payload-pre-processing" Nov 03 08:53:12.401626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401518 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38635fab-033d-4101-8467-b893aa1024e7" containerName="payload-pre-processing" Nov 03 08:53:12.401626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401535 5015 cpu_manager.go:414] "containerMap: removing container" podUID="c121a127-8ef7-4c8b-8d9f-50d457f6f265" containerName="payload-pre-processing" Nov 03 08:53:12.401626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401542 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c121a127-8ef7-4c8b-8d9f-50d457f6f265" containerName="payload-pre-processing" Nov 03 08:53:12.401626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401554 5015 cpu_manager.go:414] "containerMap: removing container" podUID="83011dd2-5a6e-4535-9dbf-a78b5a807f19" containerName="payload-pre-processing" Nov 03 08:53:12.401626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401562 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83011dd2-5a6e-4535-9dbf-a78b5a807f19" containerName="payload-pre-processing" Nov 03 08:53:12.401626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401583 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6d8ec563-49e9-4d9f-b595-f066173631b2" containerName="revoke-keys" Nov 03 08:53:12.401626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401590 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8ec563-49e9-4d9f-b595-f066173631b2" containerName="revoke-keys" Nov 03 08:53:12.401626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401602 5015 cpu_manager.go:414] "containerMap: removing container" podUID="24e84e39-42e3-4475-ad5e-8ec7d2438f60" containerName="payload-processing" Nov 03 08:53:12.401626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401611 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e84e39-42e3-4475-ad5e-8ec7d2438f60" containerName="payload-processing" Nov 03 08:53:12.402041 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401650 5015 cpu_manager.go:414] "containerMap: removing container" podUID="d1d7f676-6df9-47e6-a375-12b0e81bfdc6" containerName="payload-processing" Nov 03 08:53:12.402041 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401658 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1d7f676-6df9-47e6-a375-12b0e81bfdc6" containerName="payload-processing" Nov 03 08:53:12.402041 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401788 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1d7f676-6df9-47e6-a375-12b0e81bfdc6" containerName="payload-processing" Nov 03 08:53:12.402041 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401805 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="38635fab-033d-4101-8467-b893aa1024e7" containerName="payload-pre-processing" Nov 03 08:53:12.402041 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401815 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="c121a127-8ef7-4c8b-8d9f-50d457f6f265" containerName="payload-pre-processing" Nov 03 08:53:12.402041 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401826 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="83011dd2-5a6e-4535-9dbf-a78b5a807f19" containerName="payload-pre-processing" Nov 03 08:53:12.402041 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401837 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8ec563-49e9-4d9f-b595-f066173631b2" containerName="revoke-keys" Nov 03 08:53:12.402041 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.401846 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e84e39-42e3-4475-ad5e-8ec7d2438f60" containerName="payload-processing" Nov 03 08:53:12.406613 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.406590 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.437751 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.437719 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.437898 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.437772 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.437898 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.437871 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl46c\" (UniqueName: \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-kube-api-access-cl46c\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.438009 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.437984 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.438043 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.438031 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.438087 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.438072 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.438165 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.438147 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.438208 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.438182 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.438256 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.438225 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.465370 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.465337 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2"] Nov 03 08:53:12.512406 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.512379 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-18f9da-openshift-default-dockercfg-zmkbm\"" Nov 03 08:53:12.541246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.539868 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.541246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.539928 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.541246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.539962 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl46c\" (UniqueName: \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-kube-api-access-cl46c\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.541246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.540008 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.541246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.540036 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.541246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.540077 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.541246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.540108 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.541246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.540135 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.541246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.540280 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.541246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.540925 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.542790 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.542768 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.543542 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.543140 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.543686 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.543181 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.547032 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.543381 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.547197 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.544664 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.553583 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.553555 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.603994 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.603961 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl46c\" (UniqueName: \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-kube-api-access-cl46c\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.605375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.605353 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:12.720503 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:12.720393 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:13.224639 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:13.224617 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d8ec563-49e9-4d9f-b595-f066173631b2" path="/var/lib/kubelet/pods/6d8ec563-49e9-4d9f-b595-f066173631b2/volumes" Nov 03 08:53:13.225015 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:13.224996 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2"] Nov 03 08:53:13.226921 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:13.226885 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:53:13.227053 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:13.226971 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:53:13.227053 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:13.227014 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:53:13.299957 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:13.299927 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" event={"ID":"5ecc5d8b-4c77-448e-afd9-795bf6833b69","Type":"ContainerStarted","Data":"8f248397b5e05060415c6bd21c6f29b6da8e7e5af2968e07105ba3e4db206d7b"} Nov 03 08:53:14.307271 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:14.307233 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" event={"ID":"5ecc5d8b-4c77-448e-afd9-795bf6833b69","Type":"ContainerStarted","Data":"5b99d6605bd39957869b5ec5ff1fda856aed0f4ef0d6a7920dead0275268ce99"} Nov 03 08:53:14.433085 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:14.433032 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" podStartSLOduration=2.43301543 podStartE2EDuration="2.43301543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:14.432342805 +0000 UTC m=+827.357285042" watchObservedRunningTime="2026-11-03 08:53:14.43301543 +0000 UTC m=+827.357957716" Nov 03 08:53:14.721578 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:14.721541 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:14.727600 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:14.727575 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:15.313079 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.313051 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:15.314166 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.314150 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:15.536584 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.536549 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc"] Nov 03 08:53:15.536795 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.536763 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" podUID="0d7e59b6-313f-47bc-89ed-2e29da3ec030" containerName="maas-api" containerID="cri-o://d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b" gracePeriod=30 Nov 03 08:53:15.827099 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.827032 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:53:15.883240 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.883211 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7e59b6-313f-47bc-89ed-2e29da3ec030-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7e59b6-313f-47bc-89ed-2e29da3ec030-openshift-service-ca\") pod \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " Nov 03 08:53:15.883433 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.883246 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-maas-api-tls\") pod \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " Nov 03 08:53:15.883433 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.883275 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7e59b6-313f-47bc-89ed-2e29da3ec030-kube-api-access-cffbd\" (UniqueName: \"kubernetes.io/projected/0d7e59b6-313f-47bc-89ed-2e29da3ec030-kube-api-access-cffbd\") pod \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " Nov 03 08:53:15.883433 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.883314 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-metrics-certs\") pod \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\" (UID: \"0d7e59b6-313f-47bc-89ed-2e29da3ec030\") " Nov 03 08:53:15.883683 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.883644 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7e59b6-313f-47bc-89ed-2e29da3ec030-openshift-service-ca" pod "0d7e59b6-313f-47bc-89ed-2e29da3ec030" (UID: "0d7e59b6-313f-47bc-89ed-2e29da3ec030"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:15.885691 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.885659 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-maas-api-tls" pod "0d7e59b6-313f-47bc-89ed-2e29da3ec030" (UID: "0d7e59b6-313f-47bc-89ed-2e29da3ec030"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:53:15.885783 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.885694 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7e59b6-313f-47bc-89ed-2e29da3ec030-kube-api-access-cffbd" pod "0d7e59b6-313f-47bc-89ed-2e29da3ec030" (UID: "0d7e59b6-313f-47bc-89ed-2e29da3ec030"). InnerVolumeSpecName "kube-api-access-cffbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:15.885783 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.885725 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-metrics-certs" pod "0d7e59b6-313f-47bc-89ed-2e29da3ec030" (UID: "0d7e59b6-313f-47bc-89ed-2e29da3ec030"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:53:15.985204 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.985170 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7e59b6-313f-47bc-89ed-2e29da3ec030-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:15.985204 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.985200 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:15.985204 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.985210 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cffbd\" (UniqueName: \"kubernetes.io/projected/0d7e59b6-313f-47bc-89ed-2e29da3ec030-kube-api-access-cffbd\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:15.985525 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:15.985221 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7e59b6-313f-47bc-89ed-2e29da3ec030-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.007500 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:16.007453 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q"] Nov 03 08:53:16.107755 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:16.107667 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc"] Nov 03 08:53:16.320281 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:16.320252 5015 generic.go:356] "Generic (PLEG): container finished" podID="0d7e59b6-313f-47bc-89ed-2e29da3ec030" containerID="d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b" exitCode=0 Nov 03 08:53:16.320713 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:16.320299 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" Nov 03 08:53:16.320713 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:16.320342 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" event={"ID":"0d7e59b6-313f-47bc-89ed-2e29da3ec030","Type":"ContainerDied","Data":"d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b"} Nov 03 08:53:16.320713 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:16.320394 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc" event={"ID":"0d7e59b6-313f-47bc-89ed-2e29da3ec030","Type":"ContainerDied","Data":"ca28c0da0c9b98bc4f517a707138d51a1621b86e578a916d66f8008209f80d4c"} Nov 03 08:53:16.320713 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:16.320420 5015 scope.go:122] "RemoveContainer" containerID="d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b" Nov 03 08:53:16.331616 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:16.331580 5015 scope.go:122] "RemoveContainer" containerID="d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b" Nov 03 08:53:16.331992 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:16.331963 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b\": container with ID starting with d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b not found: ID does not exist" containerID="d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b" Nov 03 08:53:16.332064 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:16.332004 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b"} err="failed to get container status \"d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b\": rpc error: code = NotFound desc = could not find container \"d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b\": container with ID starting with d3e4fdae7db6a0605dc0548eaaa9048bc69aa148691e85fc3a92e077f214cb7b not found: ID does not exist" Nov 03 08:53:16.489522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:16.489492 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc"] Nov 03 08:53:16.495386 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:16.495360 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7e658462-594b5944cb-shbbc"] Nov 03 08:53:16.901327 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:16.901238 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:53:17.205169 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.205132 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d7e59b6-313f-47bc-89ed-2e29da3ec030" path="/var/lib/kubelet/pods/0d7e59b6-313f-47bc-89ed-2e29da3ec030/volumes" Nov 03 08:53:17.233748 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.233725 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:53:17.306279 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306250 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-socket\") pod \"5b48b991-cda9-4672-85aa-77e3c5056f59\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " Nov 03 08:53:17.306423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306295 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-token\" (UniqueName: \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-token\") pod \"5b48b991-cda9-4672-85aa-77e3c5056f59\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " Nov 03 08:53:17.306423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306339 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-kube-api-access-4dn25\" (UniqueName: \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-kube-api-access-4dn25\") pod \"5b48b991-cda9-4672-85aa-77e3c5056f59\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " Nov 03 08:53:17.306423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306357 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-credential-socket\") pod \"5b48b991-cda9-4672-85aa-77e3c5056f59\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " Nov 03 08:53:17.306423 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306378 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-envoy\") pod \"5b48b991-cda9-4672-85aa-77e3c5056f59\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " Nov 03 08:53:17.306661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306421 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-certs\") pod \"5b48b991-cda9-4672-85aa-77e3c5056f59\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " Nov 03 08:53:17.306661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306467 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b48b991-cda9-4672-85aa-77e3c5056f59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b48b991-cda9-4672-85aa-77e3c5056f59-istiod-ca-cert\") pod \"5b48b991-cda9-4672-85aa-77e3c5056f59\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " Nov 03 08:53:17.306661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306561 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-data\") pod \"5b48b991-cda9-4672-85aa-77e3c5056f59\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " Nov 03 08:53:17.306661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306576 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-socket" pod "5b48b991-cda9-4672-85aa-77e3c5056f59" (UID: "5b48b991-cda9-4672-85aa-77e3c5056f59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:17.307004 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306710 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-credential-socket" pod "5b48b991-cda9-4672-85aa-77e3c5056f59" (UID: "5b48b991-cda9-4672-85aa-77e3c5056f59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:17.307004 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306883 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-podinfo\") pod \"5b48b991-cda9-4672-85aa-77e3c5056f59\" (UID: \"5b48b991-cda9-4672-85aa-77e3c5056f59\") " Nov 03 08:53:17.307004 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306868 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-data" pod "5b48b991-cda9-4672-85aa-77e3c5056f59" (UID: "5b48b991-cda9-4672-85aa-77e3c5056f59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:17.307004 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306939 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b48b991-cda9-4672-85aa-77e3c5056f59-istiod-ca-cert" pod "5b48b991-cda9-4672-85aa-77e3c5056f59" (UID: "5b48b991-cda9-4672-85aa-77e3c5056f59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:17.307004 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.306949 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-certs" pod "5b48b991-cda9-4672-85aa-77e3c5056f59" (UID: "5b48b991-cda9-4672-85aa-77e3c5056f59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:17.307500 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.307362 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:17.307685 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.307529 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b48b991-cda9-4672-85aa-77e3c5056f59-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:17.307685 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.307546 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:17.307685 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.307560 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:17.307685 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.307574 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:17.308845 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.308824 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-token" pod "5b48b991-cda9-4672-85aa-77e3c5056f59" (UID: "5b48b991-cda9-4672-85aa-77e3c5056f59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:17.308951 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.308928 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-envoy" pod "5b48b991-cda9-4672-85aa-77e3c5056f59" (UID: "5b48b991-cda9-4672-85aa-77e3c5056f59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:17.309015 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.308998 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-kube-api-access-4dn25" pod "5b48b991-cda9-4672-85aa-77e3c5056f59" (UID: "5b48b991-cda9-4672-85aa-77e3c5056f59"). InnerVolumeSpecName "kube-api-access-4dn25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:17.309312 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.309279 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-podinfo" pod "5b48b991-cda9-4672-85aa-77e3c5056f59" (UID: "5b48b991-cda9-4672-85aa-77e3c5056f59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:53:17.329015 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.328977 5015 generic.go:356] "Generic (PLEG): container finished" podID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerID="1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6" exitCode=0 Nov 03 08:53:17.329375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.329023 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" Nov 03 08:53:17.329375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.329055 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" event={"ID":"5b48b991-cda9-4672-85aa-77e3c5056f59","Type":"ContainerDied","Data":"1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6"} Nov 03 08:53:17.329375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.329114 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc" event={"ID":"5b48b991-cda9-4672-85aa-77e3c5056f59","Type":"ContainerDied","Data":"d44d0373cf077b403ae2859ffd88837b101ffa1b1efa2b9e37881f0fa0a87ef5"} Nov 03 08:53:17.329375 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.329146 5015 scope.go:122] "RemoveContainer" containerID="1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6" Nov 03 08:53:17.339523 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.339504 5015 scope.go:122] "RemoveContainer" containerID="56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd" Nov 03 08:53:17.348748 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.348732 5015 scope.go:122] "RemoveContainer" containerID="1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6" Nov 03 08:53:17.349016 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:17.348996 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6\": container with ID starting with 1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6 not found: ID does not exist" containerID="1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6" Nov 03 08:53:17.349063 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.349026 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6"} err="failed to get container status \"1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6\": rpc error: code = NotFound desc = could not find container \"1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6\": container with ID starting with 1f91387d4f4d20edf093340a7c606f0ec7349f6b1a49daec0d31e291a8a3ccd6 not found: ID does not exist" Nov 03 08:53:17.349063 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.349042 5015 scope.go:122] "RemoveContainer" containerID="56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd" Nov 03 08:53:17.349281 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:17.349265 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd\": container with ID starting with 56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd not found: ID does not exist" containerID="56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd" Nov 03 08:53:17.349332 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.349291 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd"} err="failed to get container status \"56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd\": rpc error: code = NotFound desc = could not find container \"56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd\": container with ID starting with 56df78bb3c0739c3e7b18ef052a278455ce2f00657f91c78f191357c4e4b70bd not found: ID does not exist" Nov 03 08:53:17.408722 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.408693 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:17.408722 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.408721 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dn25\" (UniqueName: \"kubernetes.io/projected/5b48b991-cda9-4672-85aa-77e3c5056f59-kube-api-access-4dn25\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:17.408951 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.408739 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:17.408951 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.408753 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b48b991-cda9-4672-85aa-77e3c5056f59-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:17.468966 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.468884 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc"] Nov 03 08:53:17.474119 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.474090 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b6ef5ff5-openshift-default-84d9f8ffdc-l6pkc"] Nov 03 08:53:17.791963 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.791928 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th"] Nov 03 08:53:17.792432 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.792415 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" Nov 03 08:53:17.792554 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.792436 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" Nov 03 08:53:17.792554 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.792470 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0d7e59b6-313f-47bc-89ed-2e29da3ec030" containerName="maas-api" Nov 03 08:53:17.792554 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.792508 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7e59b6-313f-47bc-89ed-2e29da3ec030" containerName="maas-api" Nov 03 08:53:17.792723 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.792643 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" Nov 03 08:53:17.792723 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.792659 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7e59b6-313f-47bc-89ed-2e29da3ec030" containerName="maas-api" Nov 03 08:53:17.792723 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.792671 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" Nov 03 08:53:17.797344 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.797325 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:17.852032 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.852004 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"openshift-service-ca.crt\"" Nov 03 08:53:17.852232 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.852176 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"kube-root-ca.crt\"" Nov 03 08:53:17.852363 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.852324 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"default-dockercfg-dftgt\"" Nov 03 08:53:17.852670 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.852649 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"simulator-w3-98d5dd-kserve-self-signed-certs\"" Nov 03 08:53:17.854330 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.854309 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th"] Nov 03 08:53:17.913327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.913295 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-tmp-dir\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:17.913535 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.913433 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-home\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:17.913601 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.913556 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-dshm\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:17.913601 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.913588 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f331f4-0042-4157-9107-d5626c3fc8d4-tls-certs\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:17.913705 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.913621 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-model-cache\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:17.913705 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:17.913649 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4k82\" (UniqueName: \"kubernetes.io/projected/40f331f4-0042-4157-9107-d5626c3fc8d4-kube-api-access-r4k82\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.015201 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.015173 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-dshm\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.015201 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.015203 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f331f4-0042-4157-9107-d5626c3fc8d4-tls-certs\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.015454 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.015231 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-model-cache\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.015454 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.015255 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4k82\" (UniqueName: \"kubernetes.io/projected/40f331f4-0042-4157-9107-d5626c3fc8d4-kube-api-access-r4k82\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.015454 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.015286 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-tmp-dir\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.015454 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.015348 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-home\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.015760 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.015725 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-model-cache\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.015891 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.015765 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-home\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.015891 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.015781 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-tmp-dir\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.017730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.017703 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-dshm\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.017881 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.017865 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f331f4-0042-4157-9107-d5626c3fc8d4-tls-certs\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.080579 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.080518 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4k82\" (UniqueName: \"kubernetes.io/projected/40f331f4-0042-4157-9107-d5626c3fc8d4-kube-api-access-r4k82\") pod \"simulator-w3-98d5dd-kserve-676c669984-z59th\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.108941 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.108915 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:18.490569 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:18.490538 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th"] Nov 03 08:53:18.492151 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:53:18.492128 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40f331f4_0042_4157_9107_d5626c3fc8d4.slice/crio-cbce1a06f607899183a2b23ac6ff9b289a45bc8c68961a17ee84916251ee40ef WatchSource:0}: Error finding container cbce1a06f607899183a2b23ac6ff9b289a45bc8c68961a17ee84916251ee40ef: Status 404 returned error can't find the container with id cbce1a06f607899183a2b23ac6ff9b289a45bc8c68961a17ee84916251ee40ef Nov 03 08:53:19.203660 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:19.203631 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" path="/var/lib/kubelet/pods/5b48b991-cda9-4672-85aa-77e3c5056f59/volumes" Nov 03 08:53:19.344768 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:19.344732 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" event={"ID":"40f331f4-0042-4157-9107-d5626c3fc8d4","Type":"ContainerStarted","Data":"07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8"} Nov 03 08:53:19.344938 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:19.344780 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" event={"ID":"40f331f4-0042-4157-9107-d5626c3fc8d4","Type":"ContainerStarted","Data":"cbce1a06f607899183a2b23ac6ff9b289a45bc8c68961a17ee84916251ee40ef"} Nov 03 08:53:19.462491 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:19.462371 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podStartSLOduration=2.462355994 podStartE2EDuration="2.462355994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:19.460784954 +0000 UTC m=+832.385727171" watchObservedRunningTime="2026-11-03 08:53:19.462355994 +0000 UTC m=+832.387298212" Nov 03 08:53:20.109285 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:20.109202 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:21.008389 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.008325 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" podUID="bf81cd02-012f-44af-a044-6fa92e98f887" containerName="payload-processing" containerID="cri-o://e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284" gracePeriod=25 Nov 03 08:53:21.108425 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.108387 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" podUID="669e5187-d582-4a12-bc17-604a1d8d7b18" containerName="payload-pre-processing" containerID="cri-o://5b24dd79a641c118ba6b565c057402af7abd440322d91c86c6d4f1bb3e84d342" gracePeriod=25 Nov 03 08:53:21.285176 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.284762 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:53:21.349855 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.349825 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf81cd02-012f-44af-a044-6fa92e98f887-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf81cd02-012f-44af-a044-6fa92e98f887-plugins-config-volume\") pod \"bf81cd02-012f-44af-a044-6fa92e98f887\" (UID: \"bf81cd02-012f-44af-a044-6fa92e98f887\") " Nov 03 08:53:21.350041 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.350016 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf81cd02-012f-44af-a044-6fa92e98f887-kube-api-access-5kk6c\" (UniqueName: \"kubernetes.io/projected/bf81cd02-012f-44af-a044-6fa92e98f887-kube-api-access-5kk6c\") pod \"bf81cd02-012f-44af-a044-6fa92e98f887\" (UID: \"bf81cd02-012f-44af-a044-6fa92e98f887\") " Nov 03 08:53:21.350336 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.350289 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf81cd02-012f-44af-a044-6fa92e98f887-plugins-config-volume" pod "bf81cd02-012f-44af-a044-6fa92e98f887" (UID: "bf81cd02-012f-44af-a044-6fa92e98f887"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:21.353295 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.353265 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf81cd02-012f-44af-a044-6fa92e98f887-kube-api-access-5kk6c" pod "bf81cd02-012f-44af-a044-6fa92e98f887" (UID: "bf81cd02-012f-44af-a044-6fa92e98f887"). InnerVolumeSpecName "kube-api-access-5kk6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:21.358873 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.358846 5015 generic.go:356] "Generic (PLEG): container finished" podID="bf81cd02-012f-44af-a044-6fa92e98f887" containerID="e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284" exitCode=0 Nov 03 08:53:21.358969 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.358902 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" Nov 03 08:53:21.358969 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.358929 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" event={"ID":"bf81cd02-012f-44af-a044-6fa92e98f887","Type":"ContainerDied","Data":"e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284"} Nov 03 08:53:21.359091 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.358992 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q" event={"ID":"bf81cd02-012f-44af-a044-6fa92e98f887","Type":"ContainerDied","Data":"eb661729ce3d69d8d7ba6f1f4fb8f9ac5ba3fc7892eba98fc5ac1065e0fe7120"} Nov 03 08:53:21.359091 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.359024 5015 scope.go:122] "RemoveContainer" containerID="e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284" Nov 03 08:53:21.361747 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.361178 5015 generic.go:356] "Generic (PLEG): container finished" podID="669e5187-d582-4a12-bc17-604a1d8d7b18" containerID="5b24dd79a641c118ba6b565c057402af7abd440322d91c86c6d4f1bb3e84d342" exitCode=0 Nov 03 08:53:21.361747 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.361250 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" event={"ID":"669e5187-d582-4a12-bc17-604a1d8d7b18","Type":"ContainerDied","Data":"5b24dd79a641c118ba6b565c057402af7abd440322d91c86c6d4f1bb3e84d342"} Nov 03 08:53:21.377448 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.377426 5015 scope.go:122] "RemoveContainer" containerID="e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284" Nov 03 08:53:21.377862 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:21.377842 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284\": container with ID starting with e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284 not found: ID does not exist" containerID="e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284" Nov 03 08:53:21.377945 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.377874 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284"} err="failed to get container status \"e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284\": rpc error: code = NotFound desc = could not find container \"e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284\": container with ID starting with e602f05e8fa7028fc61c0909129b6434bb773128e3367632959794b5579f0284 not found: ID does not exist" Nov 03 08:53:21.384562 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.384542 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:53:21.451233 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.451197 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669e5187-d582-4a12-bc17-604a1d8d7b18-kube-api-access-7jrf7\" (UniqueName: \"kubernetes.io/projected/669e5187-d582-4a12-bc17-604a1d8d7b18-kube-api-access-7jrf7\") pod \"669e5187-d582-4a12-bc17-604a1d8d7b18\" (UID: \"669e5187-d582-4a12-bc17-604a1d8d7b18\") " Nov 03 08:53:21.451500 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.451259 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669e5187-d582-4a12-bc17-604a1d8d7b18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669e5187-d582-4a12-bc17-604a1d8d7b18-plugins-config-volume\") pod \"669e5187-d582-4a12-bc17-604a1d8d7b18\" (UID: \"669e5187-d582-4a12-bc17-604a1d8d7b18\") " Nov 03 08:53:21.451791 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.451767 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kk6c\" (UniqueName: \"kubernetes.io/projected/bf81cd02-012f-44af-a044-6fa92e98f887-kube-api-access-5kk6c\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:21.451867 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.451794 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf81cd02-012f-44af-a044-6fa92e98f887-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:21.451867 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.451825 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669e5187-d582-4a12-bc17-604a1d8d7b18-plugins-config-volume" pod "669e5187-d582-4a12-bc17-604a1d8d7b18" (UID: "669e5187-d582-4a12-bc17-604a1d8d7b18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:21.453653 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.453634 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669e5187-d582-4a12-bc17-604a1d8d7b18-kube-api-access-7jrf7" pod "669e5187-d582-4a12-bc17-604a1d8d7b18" (UID: "669e5187-d582-4a12-bc17-604a1d8d7b18"). InnerVolumeSpecName "kube-api-access-7jrf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:21.553627 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.553537 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jrf7\" (UniqueName: \"kubernetes.io/projected/669e5187-d582-4a12-bc17-604a1d8d7b18-kube-api-access-7jrf7\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:21.553627 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.553575 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669e5187-d582-4a12-bc17-604a1d8d7b18-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:21.637458 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.637419 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q"] Nov 03 08:53:21.672932 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.657112 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7e658462-5bfc844c7-d4v4q"] Nov 03 08:53:21.779421 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.779382 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l"] Nov 03 08:53:21.779643 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.779624 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" containerID="cri-o://929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205" gracePeriod=30 Nov 03 08:53:21.962628 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.962537 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7"] Nov 03 08:53:21.963175 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.963155 5015 cpu_manager.go:414] "containerMap: removing container" podUID="669e5187-d582-4a12-bc17-604a1d8d7b18" containerName="payload-pre-processing" Nov 03 08:53:21.963242 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.963180 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669e5187-d582-4a12-bc17-604a1d8d7b18" containerName="payload-pre-processing" Nov 03 08:53:21.963242 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.963220 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" Nov 03 08:53:21.963242 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.963228 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b48b991-cda9-4672-85aa-77e3c5056f59" containerName="istio-proxy" Nov 03 08:53:21.963386 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.963249 5015 cpu_manager.go:414] "containerMap: removing container" podUID="bf81cd02-012f-44af-a044-6fa92e98f887" containerName="payload-processing" Nov 03 08:53:21.963386 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.963257 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf81cd02-012f-44af-a044-6fa92e98f887" containerName="payload-processing" Nov 03 08:53:21.963495 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.963401 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="669e5187-d582-4a12-bc17-604a1d8d7b18" containerName="payload-pre-processing" Nov 03 08:53:21.963495 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.963422 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf81cd02-012f-44af-a044-6fa92e98f887" containerName="payload-processing" Nov 03 08:53:21.967241 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:21.967222 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 08:53:22.023323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.023294 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7"] Nov 03 08:53:22.023820 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.023797 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-18f9da\"" Nov 03 08:53:22.023961 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.023867 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-18f9da-dockercfg-n28mf\"" Nov 03 08:53:22.060371 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.060335 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d86ce37-a32d-4d80-8331-f1f0b10c0048-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7\" (UID: \"8d86ce37-a32d-4d80-8331-f1f0b10c0048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 08:53:22.060585 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.060530 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvhxk\" (UniqueName: \"kubernetes.io/projected/8d86ce37-a32d-4d80-8331-f1f0b10c0048-kube-api-access-xvhxk\") pod \"payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7\" (UID: \"8d86ce37-a32d-4d80-8331-f1f0b10c0048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 08:53:22.106634 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.106600 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94"] Nov 03 08:53:22.110703 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.110681 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 08:53:22.161849 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.161807 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhxk\" (UniqueName: \"kubernetes.io/projected/8d86ce37-a32d-4d80-8331-f1f0b10c0048-kube-api-access-xvhxk\") pod \"payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7\" (UID: \"8d86ce37-a32d-4d80-8331-f1f0b10c0048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 08:53:22.162111 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.161985 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d86ce37-a32d-4d80-8331-f1f0b10c0048-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7\" (UID: \"8d86ce37-a32d-4d80-8331-f1f0b10c0048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 08:53:22.163005 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.162959 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d86ce37-a32d-4d80-8331-f1f0b10c0048-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7\" (UID: \"8d86ce37-a32d-4d80-8331-f1f0b10c0048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 08:53:22.168716 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.168685 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94"] Nov 03 08:53:22.222988 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.222953 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhxk\" (UniqueName: \"kubernetes.io/projected/8d86ce37-a32d-4d80-8331-f1f0b10c0048-kube-api-access-xvhxk\") pod \"payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7\" (UID: \"8d86ce37-a32d-4d80-8331-f1f0b10c0048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 08:53:22.263244 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.263213 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724c937b-6a9c-4c29-93d5-4a434f384a93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94\" (UID: \"724c937b-6a9c-4c29-93d5-4a434f384a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 08:53:22.263399 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.263340 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnfxp\" (UniqueName: \"kubernetes.io/projected/724c937b-6a9c-4c29-93d5-4a434f384a93-kube-api-access-tnfxp\") pod \"payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94\" (UID: \"724c937b-6a9c-4c29-93d5-4a434f384a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 08:53:22.282193 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.282160 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 08:53:22.366702 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.366144 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnfxp\" (UniqueName: \"kubernetes.io/projected/724c937b-6a9c-4c29-93d5-4a434f384a93-kube-api-access-tnfxp\") pod \"payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94\" (UID: \"724c937b-6a9c-4c29-93d5-4a434f384a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 08:53:22.366702 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.366262 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724c937b-6a9c-4c29-93d5-4a434f384a93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94\" (UID: \"724c937b-6a9c-4c29-93d5-4a434f384a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 08:53:22.368076 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.368009 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724c937b-6a9c-4c29-93d5-4a434f384a93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94\" (UID: \"724c937b-6a9c-4c29-93d5-4a434f384a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 08:53:22.378735 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.378296 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" event={"ID":"669e5187-d582-4a12-bc17-604a1d8d7b18","Type":"ContainerDied","Data":"ce075831c0e57371577a123f63985dcd120299a76c7c2f5758a2e711bf13e6ae"} Nov 03 08:53:22.378735 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.378362 5015 scope.go:122] "RemoveContainer" containerID="5b24dd79a641c118ba6b565c057402af7abd440322d91c86c6d4f1bb3e84d342" Nov 03 08:53:22.378735 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.378551 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc" Nov 03 08:53:22.426942 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.426907 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnfxp\" (UniqueName: \"kubernetes.io/projected/724c937b-6a9c-4c29-93d5-4a434f384a93-kube-api-access-tnfxp\") pod \"payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94\" (UID: \"724c937b-6a9c-4c29-93d5-4a434f384a93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 08:53:22.452373 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.452352 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 08:53:22.553610 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.552596 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc"] Nov 03 08:53:22.557189 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.557159 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e658462-5bd7c6bb4c-98dnc"] Nov 03 08:53:22.678215 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.678190 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7"] Nov 03 08:53:22.679994 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:53:22.679966 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d86ce37_a32d_4d80_8331_f1f0b10c0048.slice/crio-e5a94f4b595657fee2dcc2db63bccf2e53bc1a8ef6490ad329d585d38e35e2d6 WatchSource:0}: Error finding container e5a94f4b595657fee2dcc2db63bccf2e53bc1a8ef6490ad329d585d38e35e2d6: Status 404 returned error can't find the container with id e5a94f4b595657fee2dcc2db63bccf2e53bc1a8ef6490ad329d585d38e35e2d6 Nov 03 08:53:22.832238 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:22.832212 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94"] Nov 03 08:53:22.833043 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:53:22.833008 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724c937b_6a9c_4c29_93d5_4a434f384a93.slice/crio-e082603a12ffbb49168a6ac34277a795ea692dc11df721adca05ba71f9255781 WatchSource:0}: Error finding container e082603a12ffbb49168a6ac34277a795ea692dc11df721adca05ba71f9255781: Status 404 returned error can't find the container with id e082603a12ffbb49168a6ac34277a795ea692dc11df721adca05ba71f9255781 Nov 03 08:53:23.203185 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:23.203109 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669e5187-d582-4a12-bc17-604a1d8d7b18" path="/var/lib/kubelet/pods/669e5187-d582-4a12-bc17-604a1d8d7b18/volumes" Nov 03 08:53:23.203470 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:23.203457 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf81cd02-012f-44af-a044-6fa92e98f887" path="/var/lib/kubelet/pods/bf81cd02-012f-44af-a044-6fa92e98f887/volumes" Nov 03 08:53:23.390657 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:23.390614 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" event={"ID":"8d86ce37-a32d-4d80-8331-f1f0b10c0048","Type":"ContainerStarted","Data":"cb15b8c76c488e41f89381d28bbc3c3b9202a541d6a8be887004632da07b259f"} Nov 03 08:53:23.391217 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:23.391192 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 08:53:23.391302 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:23.391229 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" event={"ID":"8d86ce37-a32d-4d80-8331-f1f0b10c0048","Type":"ContainerStarted","Data":"e5a94f4b595657fee2dcc2db63bccf2e53bc1a8ef6490ad329d585d38e35e2d6"} Nov 03 08:53:23.392636 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:23.392607 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" event={"ID":"724c937b-6a9c-4c29-93d5-4a434f384a93","Type":"ContainerStarted","Data":"3c838d7c6ecfd0cd7192e35c879c066c4634095a10ae366f270c9b85710db745"} Nov 03 08:53:23.392763 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:23.392653 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" event={"ID":"724c937b-6a9c-4c29-93d5-4a434f384a93","Type":"ContainerStarted","Data":"e082603a12ffbb49168a6ac34277a795ea692dc11df721adca05ba71f9255781"} Nov 03 08:53:23.392763 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:23.392681 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 08:53:23.509145 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:23.509084 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" podStartSLOduration=2.509066693 podStartE2EDuration="2.509066693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:23.5077024 +0000 UTC m=+836.432644621" watchObservedRunningTime="2026-11-03 08:53:23.509066693 +0000 UTC m=+836.434008909" Nov 03 08:53:23.628470 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:23.628395 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" podStartSLOduration=1.6283760269999998 podStartE2EDuration="1.628376027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:23.626614635 +0000 UTC m=+836.551556927" watchObservedRunningTime="2026-11-03 08:53:23.628376027 +0000 UTC m=+836.553318245" Nov 03 08:53:24.619353 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.619316 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9"] Nov 03 08:53:24.624293 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.624261 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 08:53:24.682652 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.682622 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9"] Nov 03 08:53:24.718502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.694891 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9\" (UID: \"c4df8dd8-b208-42c5-b6ef-fdc924243ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 08:53:24.718502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.695618 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dn5s\" (UniqueName: \"kubernetes.io/projected/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-kube-api-access-4dn5s\") pod \"payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9\" (UID: \"c4df8dd8-b208-42c5-b6ef-fdc924243ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 08:53:24.730077 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.729510 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-77207e\"" Nov 03 08:53:24.730077 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.729875 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-77207e-dockercfg-mcf9z\"" Nov 03 08:53:24.782529 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.782485 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v"] Nov 03 08:53:24.786313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.786288 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 08:53:24.797181 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.797154 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9\" (UID: \"c4df8dd8-b208-42c5-b6ef-fdc924243ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 08:53:24.797293 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.797198 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dn5s\" (UniqueName: \"kubernetes.io/projected/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-kube-api-access-4dn5s\") pod \"payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9\" (UID: \"c4df8dd8-b208-42c5-b6ef-fdc924243ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 08:53:24.797869 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.797850 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9\" (UID: \"c4df8dd8-b208-42c5-b6ef-fdc924243ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 08:53:24.816163 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.816130 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v"] Nov 03 08:53:24.856697 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.856670 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dn5s\" (UniqueName: \"kubernetes.io/projected/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-kube-api-access-4dn5s\") pod \"payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9\" (UID: \"c4df8dd8-b208-42c5-b6ef-fdc924243ba6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 08:53:24.898661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.898572 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7rlz\" (UniqueName: \"kubernetes.io/projected/0bbca83d-8f02-412a-b508-2633ec103a94-kube-api-access-l7rlz\") pod \"payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v\" (UID: \"0bbca83d-8f02-412a-b508-2633ec103a94\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 08:53:24.898661 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.898611 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bbca83d-8f02-412a-b508-2633ec103a94-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v\" (UID: \"0bbca83d-8f02-412a-b508-2633ec103a94\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 08:53:24.938230 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:24.938187 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 08:53:25.001094 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:25.001063 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7rlz\" (UniqueName: \"kubernetes.io/projected/0bbca83d-8f02-412a-b508-2633ec103a94-kube-api-access-l7rlz\") pod \"payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v\" (UID: \"0bbca83d-8f02-412a-b508-2633ec103a94\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 08:53:25.001233 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:25.001114 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bbca83d-8f02-412a-b508-2633ec103a94-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v\" (UID: \"0bbca83d-8f02-412a-b508-2633ec103a94\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 08:53:25.002224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:25.002198 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bbca83d-8f02-412a-b508-2633ec103a94-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v\" (UID: \"0bbca83d-8f02-412a-b508-2633ec103a94\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 08:53:25.060982 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:25.060949 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7rlz\" (UniqueName: \"kubernetes.io/projected/0bbca83d-8f02-412a-b508-2633ec103a94-kube-api-access-l7rlz\") pod \"payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v\" (UID: \"0bbca83d-8f02-412a-b508-2633ec103a94\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 08:53:25.099884 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:25.099852 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 08:53:25.314374 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:25.314345 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9"] Nov 03 08:53:25.315320 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:53:25.315294 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4df8dd8_b208_42c5_b6ef_fdc924243ba6.slice/crio-09d62d23fc8cdd56376cffe58c930655f8062b40ea00bedf68e30abd4f431ed2 WatchSource:0}: Error finding container 09d62d23fc8cdd56376cffe58c930655f8062b40ea00bedf68e30abd4f431ed2: Status 404 returned error can't find the container with id 09d62d23fc8cdd56376cffe58c930655f8062b40ea00bedf68e30abd4f431ed2 Nov 03 08:53:25.405390 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:25.405323 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" event={"ID":"c4df8dd8-b208-42c5-b6ef-fdc924243ba6","Type":"ContainerStarted","Data":"b490bd341aa36a451c32c48403eeb41c132f6600df744c7273eb31a9c51ca0c8"} Nov 03 08:53:25.405390 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:25.405378 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" event={"ID":"c4df8dd8-b208-42c5-b6ef-fdc924243ba6","Type":"ContainerStarted","Data":"09d62d23fc8cdd56376cffe58c930655f8062b40ea00bedf68e30abd4f431ed2"} Nov 03 08:53:25.405583 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:25.405468 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 08:53:25.481520 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:25.481496 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v"] Nov 03 08:53:25.481610 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:53:25.481561 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbca83d_8f02_412a_b508_2633ec103a94.slice/crio-e79ae7c7fa87e14405db27fd132303694b220423f4e53b0eb83daaffdb7559a4 WatchSource:0}: Error finding container e79ae7c7fa87e14405db27fd132303694b220423f4e53b0eb83daaffdb7559a4: Status 404 returned error can't find the container with id e79ae7c7fa87e14405db27fd132303694b220423f4e53b0eb83daaffdb7559a4 Nov 03 08:53:25.531196 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:25.531152 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" podStartSLOduration=1.531136183 podStartE2EDuration="1.531136183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:25.529104479 +0000 UTC m=+838.454046697" watchObservedRunningTime="2026-11-03 08:53:25.531136183 +0000 UTC m=+838.456078399" Nov 03 08:53:26.413959 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:26.413912 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" event={"ID":"0bbca83d-8f02-412a-b508-2633ec103a94","Type":"ContainerStarted","Data":"1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa"} Nov 03 08:53:26.414538 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:26.413986 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" event={"ID":"0bbca83d-8f02-412a-b508-2633ec103a94","Type":"ContainerStarted","Data":"e79ae7c7fa87e14405db27fd132303694b220423f4e53b0eb83daaffdb7559a4"} Nov 03 08:53:26.414538 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:26.414258 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 08:53:26.532757 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:26.532706 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" podStartSLOduration=2.532691848 podStartE2EDuration="2.532691848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:26.530268513 +0000 UTC m=+839.455210730" watchObservedRunningTime="2026-11-03 08:53:26.532691848 +0000 UTC m=+839.457634064" Nov 03 08:53:27.051577 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.051517 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:53:27.222543 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222514 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istiod-ca-cert\") pod \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " Nov 03 08:53:27.222696 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222551 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-certs\") pod \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " Nov 03 08:53:27.222696 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222592 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-credential-socket\") pod \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " Nov 03 08:53:27.222696 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222657 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-podinfo\") pod \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " Nov 03 08:53:27.222696 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222693 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-envoy\") pod \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " Nov 03 08:53:27.222903 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222737 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-kube-api-access-lkpff\" (UniqueName: \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-kube-api-access-lkpff\") pod \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " Nov 03 08:53:27.222903 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222807 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-socket\") pod \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " Nov 03 08:53:27.222903 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222814 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-certs" pod "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" (UID: "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:27.222903 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222853 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-token\" (UniqueName: \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-token\") pod \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " Nov 03 08:53:27.223120 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222912 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-data\") pod \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\" (UID: \"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96\") " Nov 03 08:53:27.223120 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222940 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-credential-socket" pod "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" (UID: "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:27.223120 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.222957 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istiod-ca-cert" pod "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" (UID: "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:27.223276 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.223206 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:27.223276 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.223228 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:27.223276 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.223240 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:27.223276 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.223210 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-data" pod "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" (UID: "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:27.223680 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.223519 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-socket" pod "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" (UID: "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:27.225003 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.224971 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-podinfo" pod "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" (UID: "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:53:27.225127 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.225045 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-envoy" pod "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" (UID: "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:27.225173 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.225131 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-token" pod "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" (UID: "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:27.225832 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.225802 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-kube-api-access-lkpff" pod "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" (UID: "081a7f81-c5fb-45f6-964e-5e2dc4e8fb96"). InnerVolumeSpecName "kube-api-access-lkpff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:27.324065 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.323981 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:27.324065 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.324010 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:27.324065 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.324019 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:27.324065 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.324033 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:27.324065 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.324045 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkpff\" (UniqueName: \"kubernetes.io/projected/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-kube-api-access-lkpff\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:27.324065 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.324053 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:27.421060 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.421029 5015 generic.go:356] "Generic (PLEG): container finished" podID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerID="929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205" exitCode=0 Nov 03 08:53:27.421429 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.421089 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" Nov 03 08:53:27.421429 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.421117 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" event={"ID":"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96","Type":"ContainerDied","Data":"929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205"} Nov 03 08:53:27.421429 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.421170 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l" event={"ID":"081a7f81-c5fb-45f6-964e-5e2dc4e8fb96","Type":"ContainerDied","Data":"137b146f321f282dc89b30d42843b0a1a353df033956c4b7b7a646dcec3c8eac"} Nov 03 08:53:27.421429 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.421203 5015 scope.go:122] "RemoveContainer" containerID="929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205" Nov 03 08:53:27.430593 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.430573 5015 scope.go:122] "RemoveContainer" containerID="fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816" Nov 03 08:53:27.438898 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.438880 5015 scope.go:122] "RemoveContainer" containerID="929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205" Nov 03 08:53:27.439154 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:27.439132 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205\": container with ID starting with 929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205 not found: ID does not exist" containerID="929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205" Nov 03 08:53:27.439224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.439166 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205"} err="failed to get container status \"929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205\": rpc error: code = NotFound desc = could not find container \"929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205\": container with ID starting with 929724262a21745c5984a14a88c033b2c1d839c6992924bfea75c7a21e006205 not found: ID does not exist" Nov 03 08:53:27.439224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.439190 5015 scope.go:122] "RemoveContainer" containerID="fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816" Nov 03 08:53:27.439453 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:27.439436 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816\": container with ID starting with fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816 not found: ID does not exist" containerID="fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816" Nov 03 08:53:27.439551 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.439459 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816"} err="failed to get container status \"fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816\": rpc error: code = NotFound desc = could not find container \"fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816\": container with ID starting with fd2dd5b1fc82e9c19dc255928276e9bd58c8f41635431d2b6b9642d09d26f816 not found: ID does not exist" Nov 03 08:53:27.592431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.592356 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l"] Nov 03 08:53:27.642618 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:27.642592 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7e658462-openshift-default-795dc55f57-2vm6l"] Nov 03 08:53:28.110076 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.110042 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:28.122799 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.122775 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:28.194368 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.194340 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9"] Nov 03 08:53:28.194830 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.194816 5015 cpu_manager.go:414] "containerMap: removing container" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" Nov 03 08:53:28.194883 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.194831 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" Nov 03 08:53:28.194883 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.194840 5015 cpu_manager.go:414] "containerMap: removing container" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" Nov 03 08:53:28.194883 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.194845 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" Nov 03 08:53:28.194971 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.194928 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" Nov 03 08:53:28.194971 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.194937 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" containerName="istio-proxy" Nov 03 08:53:28.200219 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.200193 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.254959 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.254936 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-18f9da-serving-cert\"" Nov 03 08:53:28.255118 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.254986 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 08:53:28.256633 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.256610 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9"] Nov 03 08:53:28.290463 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.290435 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f42g7\"" Nov 03 08:53:28.335051 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.335014 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.335226 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.335167 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4sct\" (UniqueName: \"kubernetes.io/projected/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-kube-api-access-m4sct\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.335304 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.335260 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.335373 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.335321 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-metrics-certs\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.436382 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.436303 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.436382 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.436348 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4sct\" (UniqueName: \"kubernetes.io/projected/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-kube-api-access-m4sct\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.436897 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.436405 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.436897 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:28.436431 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-serving-cert: secret "maas-api-e2e-worker-w4-18f9da-serving-cert" not found Nov 03 08:53:28.436897 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.436471 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-metrics-certs\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.436897 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:28.436528 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-maas-api-tls podName:ed11b2ca-3e57-40d1-8eeb-20a31f9533c2 nodeName:}" failed. No retries permitted until 2026-11-03 08:53:28.936508196 +0000 UTC m=+841.861450417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-maas-api-tls") pod "maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" (UID: "ed11b2ca-3e57-40d1-8eeb-20a31f9533c2") : secret "maas-api-e2e-worker-w4-18f9da-serving-cert" not found Nov 03 08:53:28.437105 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.437087 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.439090 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.439070 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-metrics-certs\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.453207 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.453162 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 08:53:28.503304 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.503270 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4sct\" (UniqueName: \"kubernetes.io/projected/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-kube-api-access-m4sct\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.942569 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.942525 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:28.945188 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:28.945164 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:29.113566 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:29.113535 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:29.208003 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:29.207912 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081a7f81-c5fb-45f6-964e-5e2dc4e8fb96" path="/var/lib/kubelet/pods/081a7f81-c5fb-45f6-964e-5e2dc4e8fb96/volumes" Nov 03 08:53:29.539462 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:29.539429 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9"] Nov 03 08:53:29.540781 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:53:29.540744 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded11b2ca_3e57_40d1_8eeb_20a31f9533c2.slice/crio-d614e62e473faf3a336424b4ed47cb179f2d4fc531393eb1df7321637d6b9e71 WatchSource:0}: Error finding container d614e62e473faf3a336424b4ed47cb179f2d4fc531393eb1df7321637d6b9e71: Status 404 returned error can't find the container with id d614e62e473faf3a336424b4ed47cb179f2d4fc531393eb1df7321637d6b9e71 Nov 03 08:53:30.450156 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:30.450132 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" event={"ID":"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2","Type":"ContainerStarted","Data":"d614e62e473faf3a336424b4ed47cb179f2d4fc531393eb1df7321637d6b9e71"} Nov 03 08:53:31.459658 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:31.459621 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" event={"ID":"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2","Type":"ContainerStarted","Data":"97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924"} Nov 03 08:53:31.460341 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:31.460321 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:31.578322 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:31.578265 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" podStartSLOduration=2.746812625 podStartE2EDuration="3.578250452s" totalImagesPullingTime="831.437827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:31.577219259 +0000 UTC m=+844.502161480" watchObservedRunningTime="2026-11-03 08:53:31.578250452 +0000 UTC m=+844.503192669" Nov 03 08:53:31.946306 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:31.946276 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4"] Nov 03 08:53:31.954349 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:31.954325 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 08:53:32.009447 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.009400 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-26e861\"" Nov 03 08:53:32.012294 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.012268 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4"] Nov 03 08:53:32.067545 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.067513 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-26e861-dockercfg-7smhr\"" Nov 03 08:53:32.076796 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.076768 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe094fc-cbb3-4e65-9f1f-2fc328584884-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4\" (UID: \"efe094fc-cbb3-4e65-9f1f-2fc328584884\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 08:53:32.076988 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.076905 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sb5w\" (UniqueName: \"kubernetes.io/projected/efe094fc-cbb3-4e65-9f1f-2fc328584884-kube-api-access-5sb5w\") pod \"payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4\" (UID: \"efe094fc-cbb3-4e65-9f1f-2fc328584884\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 08:53:32.089491 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.089451 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l"] Nov 03 08:53:32.093650 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.093632 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 08:53:32.152270 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.152238 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l"] Nov 03 08:53:32.177977 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.177948 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sb5w\" (UniqueName: \"kubernetes.io/projected/efe094fc-cbb3-4e65-9f1f-2fc328584884-kube-api-access-5sb5w\") pod \"payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4\" (UID: \"efe094fc-cbb3-4e65-9f1f-2fc328584884\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 08:53:32.178147 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.178043 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe094fc-cbb3-4e65-9f1f-2fc328584884-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4\" (UID: \"efe094fc-cbb3-4e65-9f1f-2fc328584884\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 08:53:32.178821 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.178793 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe094fc-cbb3-4e65-9f1f-2fc328584884-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4\" (UID: \"efe094fc-cbb3-4e65-9f1f-2fc328584884\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 08:53:32.237676 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.237640 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sb5w\" (UniqueName: \"kubernetes.io/projected/efe094fc-cbb3-4e65-9f1f-2fc328584884-kube-api-access-5sb5w\") pod \"payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4\" (UID: \"efe094fc-cbb3-4e65-9f1f-2fc328584884\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 08:53:32.268328 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.268293 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 08:53:32.278913 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.278888 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff24799-01ca-45f0-8370-b5158f0c258a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l\" (UID: \"eff24799-01ca-45f0-8370-b5158f0c258a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 08:53:32.279044 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.279020 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6t47\" (UniqueName: \"kubernetes.io/projected/eff24799-01ca-45f0-8370-b5158f0c258a-kube-api-access-b6t47\") pod \"payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l\" (UID: \"eff24799-01ca-45f0-8370-b5158f0c258a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 08:53:32.379779 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.379735 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6t47\" (UniqueName: \"kubernetes.io/projected/eff24799-01ca-45f0-8370-b5158f0c258a-kube-api-access-b6t47\") pod \"payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l\" (UID: \"eff24799-01ca-45f0-8370-b5158f0c258a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 08:53:32.380189 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.380159 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff24799-01ca-45f0-8370-b5158f0c258a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l\" (UID: \"eff24799-01ca-45f0-8370-b5158f0c258a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 08:53:32.380915 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.380892 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff24799-01ca-45f0-8370-b5158f0c258a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l\" (UID: \"eff24799-01ca-45f0-8370-b5158f0c258a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 08:53:32.439695 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.439659 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6t47\" (UniqueName: \"kubernetes.io/projected/eff24799-01ca-45f0-8370-b5158f0c258a-kube-api-access-b6t47\") pod \"payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l\" (UID: \"eff24799-01ca-45f0-8370-b5158f0c258a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 08:53:32.644765 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.644734 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4"] Nov 03 08:53:32.646030 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:53:32.646003 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe094fc_cbb3_4e65_9f1f_2fc328584884.slice/crio-49d340e1dc5678a3eba9c0b0775268d93d52eaaa8218280f7e03aa4131a41faa WatchSource:0}: Error finding container 49d340e1dc5678a3eba9c0b0775268d93d52eaaa8218280f7e03aa4131a41faa: Status 404 returned error can't find the container with id 49d340e1dc5678a3eba9c0b0775268d93d52eaaa8218280f7e03aa4131a41faa Nov 03 08:53:32.707590 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:32.707566 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 08:53:33.087804 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:33.087771 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l"] Nov 03 08:53:33.088749 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:53:33.088721 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff24799_01ca_45f0_8370_b5158f0c258a.slice/crio-c3c1e277de91b10a9db49f1b3e7d0c300aeefbe75b155437d64a7fba500d2a67 WatchSource:0}: Error finding container c3c1e277de91b10a9db49f1b3e7d0c300aeefbe75b155437d64a7fba500d2a67: Status 404 returned error can't find the container with id c3c1e277de91b10a9db49f1b3e7d0c300aeefbe75b155437d64a7fba500d2a67 Nov 03 08:53:33.474184 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:33.474152 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" event={"ID":"eff24799-01ca-45f0-8370-b5158f0c258a","Type":"ContainerStarted","Data":"bed54b8ac55c1d7ba18d1e1685a2882c8e4901e9d242215243ba911b54b5e8a9"} Nov 03 08:53:33.474371 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:33.474206 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" event={"ID":"eff24799-01ca-45f0-8370-b5158f0c258a","Type":"ContainerStarted","Data":"c3c1e277de91b10a9db49f1b3e7d0c300aeefbe75b155437d64a7fba500d2a67"} Nov 03 08:53:33.474371 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:33.474256 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 08:53:33.476007 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:33.475981 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" event={"ID":"efe094fc-cbb3-4e65-9f1f-2fc328584884","Type":"ContainerStarted","Data":"791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233"} Nov 03 08:53:33.476153 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:33.476022 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" event={"ID":"efe094fc-cbb3-4e65-9f1f-2fc328584884","Type":"ContainerStarted","Data":"49d340e1dc5678a3eba9c0b0775268d93d52eaaa8218280f7e03aa4131a41faa"} Nov 03 08:53:33.476153 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:33.476048 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 08:53:33.590602 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:33.590552 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" podStartSLOduration=1.590536738 podStartE2EDuration="1.590536738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:33.589452691 +0000 UTC m=+846.514394919" watchObservedRunningTime="2026-11-03 08:53:33.590536738 +0000 UTC m=+846.515478972" Nov 03 08:53:33.708853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:33.708207 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" podStartSLOduration=2.708185071 podStartE2EDuration="2.708185071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:33.70463365 +0000 UTC m=+846.629575886" watchObservedRunningTime="2026-11-03 08:53:33.708185071 +0000 UTC m=+846.633127287" Nov 03 08:53:34.401539 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:34.401496 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 08:53:34.401824 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:34.401807 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 08:53:36.417827 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:36.417751 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 08:53:37.053976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.053937 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l"] Nov 03 08:53:37.058555 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.058525 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 08:53:37.115512 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.115468 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l"] Nov 03 08:53:37.145565 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.145532 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5klm\" (UniqueName: \"kubernetes.io/projected/1c5b9e02-d908-4819-8d82-f2ce501928a7-kube-api-access-p5klm\") pod \"payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l\" (UID: \"1c5b9e02-d908-4819-8d82-f2ce501928a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 08:53:37.145727 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.145661 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5b9e02-d908-4819-8d82-f2ce501928a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l\" (UID: \"1c5b9e02-d908-4819-8d82-f2ce501928a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 08:53:37.161919 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.161892 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-910d50-dockercfg-484nn\"" Nov 03 08:53:37.161919 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.161913 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-910d50\"" Nov 03 08:53:37.246274 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.246241 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5klm\" (UniqueName: \"kubernetes.io/projected/1c5b9e02-d908-4819-8d82-f2ce501928a7-kube-api-access-p5klm\") pod \"payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l\" (UID: \"1c5b9e02-d908-4819-8d82-f2ce501928a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 08:53:37.246457 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.246319 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5b9e02-d908-4819-8d82-f2ce501928a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l\" (UID: \"1c5b9e02-d908-4819-8d82-f2ce501928a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 08:53:37.247055 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.247036 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5b9e02-d908-4819-8d82-f2ce501928a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l\" (UID: \"1c5b9e02-d908-4819-8d82-f2ce501928a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 08:53:37.307167 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.307105 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5klm\" (UniqueName: \"kubernetes.io/projected/1c5b9e02-d908-4819-8d82-f2ce501928a7-kube-api-access-p5klm\") pod \"payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l\" (UID: \"1c5b9e02-d908-4819-8d82-f2ce501928a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 08:53:37.373901 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.373868 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 08:53:37.426211 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.426180 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 08:53:37.752984 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:37.752956 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l"] Nov 03 08:53:37.753197 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:53:37.753174 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5b9e02_d908_4819_8d82_f2ce501928a7.slice/crio-29c99039053bf835f3329e5eb1570876373ec926202d75dda6d5cb10297d88e5 WatchSource:0}: Error finding container 29c99039053bf835f3329e5eb1570876373ec926202d75dda6d5cb10297d88e5: Status 404 returned error can't find the container with id 29c99039053bf835f3329e5eb1570876373ec926202d75dda6d5cb10297d88e5 Nov 03 08:53:38.481244 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:38.481181 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 08:53:38.508698 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:38.508662 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" event={"ID":"1c5b9e02-d908-4819-8d82-f2ce501928a7","Type":"ContainerStarted","Data":"1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139"} Nov 03 08:53:38.508841 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:38.508721 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" event={"ID":"1c5b9e02-d908-4819-8d82-f2ce501928a7","Type":"ContainerStarted","Data":"29c99039053bf835f3329e5eb1570876373ec926202d75dda6d5cb10297d88e5"} Nov 03 08:53:38.508841 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:38.508795 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 08:53:38.717243 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:38.717176 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" podStartSLOduration=1.7171567699999999 podStartE2EDuration="1.71715677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:38.714453935 +0000 UTC m=+851.639396176" watchObservedRunningTime="2026-11-03 08:53:38.71715677 +0000 UTC m=+851.642098981" Nov 03 08:53:44.487245 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:44.487216 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 08:53:44.487765 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:44.487283 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 08:53:46.900927 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:53:46.900889 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:53:47.616277 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.616239 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76"] Nov 03 08:53:47.620914 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.620896 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.680742 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.680711 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.680742 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.680745 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b530646-50c4-4a45-9abe-35b786de0592-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.680968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.680828 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.680968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.680860 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.680968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.680875 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b530646-50c4-4a45-9abe-35b786de0592-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.680968 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.680912 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.681171 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.680976 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.681171 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.681066 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.681171 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.681102 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqck\" (UniqueName: \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-kube-api-access-nlqck\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.724815 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.724785 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-98d5dd-openshift-default-dockercfg-dt94n\"" Nov 03 08:53:47.727798 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.727771 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76"] Nov 03 08:53:47.781892 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.781799 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.781892 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.781833 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.781892 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.781867 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.782163 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.781897 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqck\" (UniqueName: \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-kube-api-access-nlqck\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.782163 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.781928 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.782163 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.781950 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b530646-50c4-4a45-9abe-35b786de0592-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.782163 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.781999 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.782163 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.782050 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.782163 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.782071 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b530646-50c4-4a45-9abe-35b786de0592-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.782453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.782263 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.782453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.782280 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.782453 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.782342 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.782687 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.782665 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.782832 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.782815 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b530646-50c4-4a45-9abe-35b786de0592-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.785189 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.785159 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.785350 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.785331 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b530646-50c4-4a45-9abe-35b786de0592-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.841786 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.841757 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqck\" (UniqueName: \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-kube-api-access-nlqck\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.841929 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.841824 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:47.939580 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:47.939548 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:48.377115 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:48.377083 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76"] Nov 03 08:53:48.377995 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:53:48.377962 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b530646_50c4_4a45_9abe_35b786de0592.slice/crio-09c0ce5ddd15a9554fcb88e9b4e4364b1a60f9c44c7e8d82c331d353348ffeb8 WatchSource:0}: Error finding container 09c0ce5ddd15a9554fcb88e9b4e4364b1a60f9c44c7e8d82c331d353348ffeb8: Status 404 returned error can't find the container with id 09c0ce5ddd15a9554fcb88e9b4e4364b1a60f9c44c7e8d82c331d353348ffeb8 Nov 03 08:53:48.380313 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:48.380278 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:53:48.380416 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:48.380341 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:53:48.380416 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:48.380370 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:53:48.598781 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:48.598742 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" event={"ID":"8b530646-50c4-4a45-9abe-35b786de0592","Type":"ContainerStarted","Data":"8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632"} Nov 03 08:53:48.598954 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:48.598803 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" event={"ID":"8b530646-50c4-4a45-9abe-35b786de0592","Type":"ContainerStarted","Data":"09c0ce5ddd15a9554fcb88e9b4e4364b1a60f9c44c7e8d82c331d353348ffeb8"} Nov 03 08:53:48.720654 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:48.720534 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" podStartSLOduration=1.72051577 podStartE2EDuration="1.72051577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:48.717895342 +0000 UTC m=+861.642837559" watchObservedRunningTime="2026-11-03 08:53:48.72051577 +0000 UTC m=+861.645457986" Nov 03 08:53:48.942648 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:48.940248 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:49.519594 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:49.519566 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 08:53:49.945651 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:49.945572 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 08:53:49.946225 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:49.946191 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" podUID="8b530646-50c4-4a45-9abe-35b786de0592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 08:53:50.947501 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:50.940881 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 08:53:50.947501 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:50.940944 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" podUID="8b530646-50c4-4a45-9abe-35b786de0592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 08:53:51.618019 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:51.617937 5015 generic.go:356] "Generic (PLEG): container finished" podID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerID="5b99d6605bd39957869b5ec5ff1fda856aed0f4ef0d6a7920dead0275268ce99" exitCode=0 Nov 03 08:53:51.618019 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:51.617992 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" event={"ID":"5ecc5d8b-4c77-448e-afd9-795bf6833b69","Type":"ContainerDied","Data":"5b99d6605bd39957869b5ec5ff1fda856aed0f4ef0d6a7920dead0275268ce99"} Nov 03 08:53:51.618363 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:51.618349 5015 scope.go:122] "RemoveContainer" containerID="5b99d6605bd39957869b5ec5ff1fda856aed0f4ef0d6a7920dead0275268ce99" Nov 03 08:53:51.619714 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:51.619685 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:53:51.619776 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:51.619755 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:53:51.619810 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:51.619788 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:53:51.941558 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:51.940876 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 08:53:51.941558 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:51.940936 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" podUID="8b530646-50c4-4a45-9abe-35b786de0592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 08:53:52.682539 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:52.671902 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" event={"ID":"5ecc5d8b-4c77-448e-afd9-795bf6833b69","Type":"ContainerStarted","Data":"48bc24f1d4e4bd7fbd01bf538b32218a0603fabd534f1589ae57560a605d39be"} Nov 03 08:53:52.720616 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:52.720568 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:52.720898 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:52.720863 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 08:53:52.721012 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:52.720927 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 08:53:52.942560 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:52.940631 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 08:53:52.942560 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:52.940688 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" podUID="8b530646-50c4-4a45-9abe-35b786de0592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 08:53:53.721833 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:53.721790 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 08:53:53.722233 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:53.721855 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 08:53:53.957943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:53.957902 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 08:53:53.959434 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:53.959402 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" podUID="8b530646-50c4-4a45-9abe-35b786de0592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 08:53:54.720870 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:54.720830 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 08:53:54.721053 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:54.720921 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 08:53:55.721581 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:55.721542 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 08:53:55.721976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:55.721595 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 08:53:55.927164 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:55.927118 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:55.927638 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:55.927610 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:55.928529 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:55.928509 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 08:53:56.721693 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:56.721660 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 08:53:56.722091 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:56.721718 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 08:53:57.720844 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:57.720812 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 08:53:57.721408 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:57.720862 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 08:53:58.726839 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:58.726811 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:58.727349 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:58.727327 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:53:58.728231 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:53:58.728207 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 08:54:16.905692 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:54:16.905328 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:54:27.351925 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:27.351894 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 08:54:27.353617 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:27.353588 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 08:54:46.902861 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:54:46.902814 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:54:54.278779 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.278743 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb"] Nov 03 08:54:54.282852 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.282830 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.387370 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.387338 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26e861\"/\"default-dockercfg-hl2h5\"" Nov 03 08:54:54.387370 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.387344 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-26e861\"/\"kube-root-ca.crt\"" Nov 03 08:54:54.387370 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.387346 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-26e861\"/\"openshift-service-ca.crt\"" Nov 03 08:54:54.387714 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.387577 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26e861\"/\"simulator-w1-26e861-kserve-self-signed-certs\"" Nov 03 08:54:54.388980 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.388958 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk296\" (UniqueName: \"kubernetes.io/projected/fa5a15cb-ba0a-463c-81cb-e85ad047b627-kube-api-access-gk296\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.389106 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.388984 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-dshm\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.389106 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.389021 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tmp-dir\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.389214 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.389149 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tls-certs\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.389214 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.389197 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-home\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.389339 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.389321 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-model-cache\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.390626 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.390589 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb"] Nov 03 08:54:54.490589 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.490552 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tls-certs\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.490809 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.490635 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-home\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.490809 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.490714 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-model-cache\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.490809 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.490769 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk296\" (UniqueName: \"kubernetes.io/projected/fa5a15cb-ba0a-463c-81cb-e85ad047b627-kube-api-access-gk296\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.490809 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.490795 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-dshm\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.491011 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.490846 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tmp-dir\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.491144 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.491122 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-home\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.491246 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.491162 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-model-cache\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.491348 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.491325 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tmp-dir\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.493208 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.493185 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-dshm\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.493303 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.493261 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tls-certs\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.551082 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.551010 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk296\" (UniqueName: \"kubernetes.io/projected/fa5a15cb-ba0a-463c-81cb-e85ad047b627-kube-api-access-gk296\") pod \"simulator-w1-26e861-kserve-5b95ccf86d-qvstb\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.596007 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.595976 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:54:54.983170 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:54.983143 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb"] Nov 03 08:54:54.985220 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:54:54.985192 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa5a15cb_ba0a_463c_81cb_e85ad047b627.slice/crio-23939eda6367eb2a5140731aac2672487f1f39ea9156f7668a6f3160585ec9bf WatchSource:0}: Error finding container 23939eda6367eb2a5140731aac2672487f1f39ea9156f7668a6f3160585ec9bf: Status 404 returned error can't find the container with id 23939eda6367eb2a5140731aac2672487f1f39ea9156f7668a6f3160585ec9bf Nov 03 08:54:55.098489 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:55.098449 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" event={"ID":"fa5a15cb-ba0a-463c-81cb-e85ad047b627","Type":"ContainerStarted","Data":"4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9"} Nov 03 08:54:55.098633 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:55.098508 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" event={"ID":"fa5a15cb-ba0a-463c-81cb-e85ad047b627","Type":"ContainerStarted","Data":"23939eda6367eb2a5140731aac2672487f1f39ea9156f7668a6f3160585ec9bf"} Nov 03 08:54:55.219310 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:55.219257 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podStartSLOduration=1.219244097 podStartE2EDuration="1.219244097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:54:55.217545657 +0000 UTC m=+928.142487896" watchObservedRunningTime="2026-11-03 08:54:55.219244097 +0000 UTC m=+928.144186369" Nov 03 08:54:55.596567 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:54:55.596535 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:55:04.597183 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:04.597148 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:55:04.610637 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:04.610605 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:55:05.176129 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:05.176046 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 08:55:16.902610 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:55:16.902567 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:55:39.210407 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.210374 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654"] Nov 03 08:55:39.214311 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.214292 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.297296 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.297258 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-dshm\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.297464 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.297310 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tmp-dir\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.297464 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.297337 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kkz4\" (UniqueName: \"kubernetes.io/projected/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-kube-api-access-2kkz4\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.297577 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.297505 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-home\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.297613 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.297601 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-model-cache\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.297646 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.297629 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tls-certs\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.318211 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.318180 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"openshift-service-ca.crt\"" Nov 03 08:55:39.318367 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.318297 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"kube-root-ca.crt\"" Nov 03 08:55:39.318431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.318359 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"default-dockercfg-czmfl\"" Nov 03 08:55:39.318431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.318378 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"premium-w2-77207e-kserve-self-signed-certs\"" Nov 03 08:55:39.318946 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.318925 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654"] Nov 03 08:55:39.398936 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.398901 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-model-cache\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.399134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.398958 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tls-certs\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.399134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.398981 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-dshm\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.399134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.399016 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tmp-dir\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.399134 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.399086 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kkz4\" (UniqueName: \"kubernetes.io/projected/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-kube-api-access-2kkz4\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.399335 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.399206 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-home\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.399419 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.399404 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-model-cache\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.399507 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.399422 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tmp-dir\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.399611 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.399592 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-home\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.401673 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.401652 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-dshm\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.401863 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.401850 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tls-certs\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.458904 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.458869 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kkz4\" (UniqueName: \"kubernetes.io/projected/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-kube-api-access-2kkz4\") pod \"premium-w2-77207e-kserve-77d794f5d4-sl654\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.526962 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.526935 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:39.912078 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:39.912055 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654"] Nov 03 08:55:39.913933 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:55:39.913898 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fafe9e0_e3d3_43d2_961a_1fd7eb9136bc.slice/crio-7af25e8c1b2f14e92be77f92ceb0972083f61e3ac1abefc4cd2b48d2fe4b2830 WatchSource:0}: Error finding container 7af25e8c1b2f14e92be77f92ceb0972083f61e3ac1abefc4cd2b48d2fe4b2830: Status 404 returned error can't find the container with id 7af25e8c1b2f14e92be77f92ceb0972083f61e3ac1abefc4cd2b48d2fe4b2830 Nov 03 08:55:40.424403 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:40.424367 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" event={"ID":"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc","Type":"ContainerStarted","Data":"b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5"} Nov 03 08:55:40.424834 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:40.424413 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" event={"ID":"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc","Type":"ContainerStarted","Data":"7af25e8c1b2f14e92be77f92ceb0972083f61e3ac1abefc4cd2b48d2fe4b2830"} Nov 03 08:55:40.531646 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:40.531337 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:40.544655 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:40.544504 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podStartSLOduration=1.544455507 podStartE2EDuration="1.544455507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:55:40.542843299 +0000 UTC m=+973.467785535" watchObservedRunningTime="2026-11-03 08:55:40.544455507 +0000 UTC m=+973.469397728" Nov 03 08:55:46.902726 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:55:46.902673 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:55:49.528069 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:49.528026 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:49.540689 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:49.540660 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:55:50.505448 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:55:50.505407 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 08:56:02.926123 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:02.926086 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72"] Nov 03 08:56:02.931061 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:02.931044 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:02.987307 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:02.987275 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72"] Nov 03 08:56:03.034624 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.034591 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26e861\"/\"premium-w1-26e861-kserve-self-signed-certs\"" Nov 03 08:56:03.067205 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.067152 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-dshm\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.079103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.067265 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-model-cache\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.079103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.074928 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-tmp-dir\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.079103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.075023 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-home\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.079103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.075377 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbfj\" (UniqueName: \"kubernetes.io/projected/0b58da17-7e7e-4a49-8550-23803cb72517-kube-api-access-phbfj\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.079103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.075448 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b58da17-7e7e-4a49-8550-23803cb72517-tls-certs\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.176581 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.176468 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-model-cache\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.176581 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.176530 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-tmp-dir\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.176819 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.176592 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-home\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.176819 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.176637 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbfj\" (UniqueName: \"kubernetes.io/projected/0b58da17-7e7e-4a49-8550-23803cb72517-kube-api-access-phbfj\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.176819 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.176688 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b58da17-7e7e-4a49-8550-23803cb72517-tls-certs\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.176819 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.176751 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-dshm\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.176986 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.176937 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-model-cache\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.177039 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.177003 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-tmp-dir\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.177093 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.177051 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-home\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.179299 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.179275 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-dshm\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.179588 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.179570 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b58da17-7e7e-4a49-8550-23803cb72517-tls-certs\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.237569 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.237530 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbfj\" (UniqueName: \"kubernetes.io/projected/0b58da17-7e7e-4a49-8550-23803cb72517-kube-api-access-phbfj\") pod \"premium-w1-26e861-kserve-5bbdc49f65-tpd72\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.245038 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.245014 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:03.640605 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:03.640580 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72"] Nov 03 08:56:03.642181 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:56:03.642153 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b58da17_7e7e_4a49_8550_23803cb72517.slice/crio-7d8d36f0d3439b9ce1b70e3fe849dc591394369aa0c263eab7428ebc6c4ee45e WatchSource:0}: Error finding container 7d8d36f0d3439b9ce1b70e3fe849dc591394369aa0c263eab7428ebc6c4ee45e: Status 404 returned error can't find the container with id 7d8d36f0d3439b9ce1b70e3fe849dc591394369aa0c263eab7428ebc6c4ee45e Nov 03 08:56:04.588733 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:04.588685 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" event={"ID":"0b58da17-7e7e-4a49-8550-23803cb72517","Type":"ContainerStarted","Data":"daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136"} Nov 03 08:56:04.588733 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:04.588734 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" event={"ID":"0b58da17-7e7e-4a49-8550-23803cb72517","Type":"ContainerStarted","Data":"7d8d36f0d3439b9ce1b70e3fe849dc591394369aa0c263eab7428ebc6c4ee45e"} Nov 03 08:56:04.706404 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:04.706346 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podStartSLOduration=2.706331928 podStartE2EDuration="2.706331928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:56:04.704587592 +0000 UTC m=+997.629529810" watchObservedRunningTime="2026-11-03 08:56:04.706331928 +0000 UTC m=+997.631274145" Nov 03 08:56:05.245634 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:05.245605 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:13.245889 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:13.245854 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:13.258465 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:13.258440 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:13.662018 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:13.661946 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 08:56:16.901970 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:56:16.901934 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:56:18.183307 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:18.183272 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6"] Nov 03 08:56:18.183750 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:18.183518 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" podUID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" containerName="istio-proxy" containerID="cri-o://8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1" gracePeriod=30 Nov 03 08:56:23.447884 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.447810 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:56:23.504361 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504325 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-certs\") pod \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " Nov 03 08:56:23.504532 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504368 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-kube-api-access-d4xzf\" (UniqueName: \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-kube-api-access-d4xzf\") pod \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " Nov 03 08:56:23.504532 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504426 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istiod-ca-cert\") pod \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " Nov 03 08:56:23.504532 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504460 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-podinfo\") pod \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " Nov 03 08:56:23.504532 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504515 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-token\" (UniqueName: \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-token\") pod \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " Nov 03 08:56:23.504749 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504551 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-envoy\") pod \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " Nov 03 08:56:23.504749 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504620 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-data\") pod \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " Nov 03 08:56:23.504749 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504622 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-certs" pod "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" (UID: "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:56:23.504749 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504706 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-credential-socket\") pod \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " Nov 03 08:56:23.504953 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504758 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-socket\") pod \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\" (UID: \"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f\") " Nov 03 08:56:23.504953 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504852 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-data" pod "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" (UID: "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:56:23.505138 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.504965 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-credential-socket" pod "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" (UID: "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:56:23.505273 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.505209 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istiod-ca-cert" pod "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" (UID: "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:56:23.505273 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.505184 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-socket" pod "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" (UID: "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:56:23.505512 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.505454 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.505512 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.505495 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.505619 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.505514 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.505619 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.505529 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.505619 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.505543 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.506915 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.506891 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-token" pod "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" (UID: "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:56:23.507005 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.506956 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-kube-api-access-d4xzf" pod "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" (UID: "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f"). InnerVolumeSpecName "kube-api-access-d4xzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:56:23.507258 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.507239 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-envoy" pod "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" (UID: "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:56:23.507854 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.507832 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-podinfo" pod "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" (UID: "58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:56:23.606397 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.606367 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4xzf\" (UniqueName: \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-kube-api-access-d4xzf\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.606397 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.606393 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.606595 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.606407 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.606595 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.606415 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.727639 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.727609 5015 generic.go:356] "Generic (PLEG): container finished" podID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" containerID="8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1" exitCode=0 Nov 03 08:56:23.727932 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.727658 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" Nov 03 08:56:23.727932 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.727705 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" event={"ID":"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f","Type":"ContainerDied","Data":"8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1"} Nov 03 08:56:23.727932 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.727766 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6" event={"ID":"58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f","Type":"ContainerDied","Data":"c90848a0479113e90963d424a6a2ac4292cef255f180026fba40e6e6e5b3d6ca"} Nov 03 08:56:23.727932 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.727798 5015 scope.go:122] "RemoveContainer" containerID="8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1" Nov 03 08:56:23.740763 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.740746 5015 scope.go:122] "RemoveContainer" containerID="8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1" Nov 03 08:56:23.741083 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:56:23.741061 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1\": container with ID starting with 8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1 not found: ID does not exist" containerID="8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1" Nov 03 08:56:23.741184 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.741095 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1"} err="failed to get container status \"8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1\": rpc error: code = NotFound desc = could not find container \"8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1\": container with ID starting with 8876c05a1eff9baeec84936a82b365d77b1a27baae2c596499f5f267619a95e1 not found: ID does not exist" Nov 03 08:56:23.899544 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.899506 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6"] Nov 03 08:56:23.904294 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:23.904268 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9n5v6"] Nov 03 08:56:24.792030 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.791999 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8"] Nov 03 08:56:24.792597 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.792583 5015 cpu_manager.go:414] "containerMap: removing container" podUID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" containerName="istio-proxy" Nov 03 08:56:24.792649 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.792599 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" containerName="istio-proxy" Nov 03 08:56:24.792700 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.792692 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" containerName="istio-proxy" Nov 03 08:56:24.797228 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.797211 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:24.819482 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.819454 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-746ab303-8qpf8\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:24.819609 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.819537 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-746ab303-8qpf8\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:24.819663 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.819621 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9kmm\" (UniqueName: \"kubernetes.io/projected/a5552d74-e374-456b-9d6b-8b4d1afb58f5-kube-api-access-q9kmm\") pod \"maas-api-revoke-sub-746ab303-8qpf8\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:24.853067 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.853042 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8"] Nov 03 08:56:24.899737 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.899705 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 08:56:24.899890 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.899744 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 08:56:24.920744 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.920712 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-746ab303-8qpf8\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:24.920883 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.920784 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-746ab303-8qpf8\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:24.920883 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.920842 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9kmm\" (UniqueName: \"kubernetes.io/projected/a5552d74-e374-456b-9d6b-8b4d1afb58f5-kube-api-access-q9kmm\") pod \"maas-api-revoke-sub-746ab303-8qpf8\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:24.921502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.921463 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-746ab303-8qpf8\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:24.923332 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.923302 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-746ab303-8qpf8\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:24.981750 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:24.981721 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9kmm\" (UniqueName: \"kubernetes.io/projected/a5552d74-e374-456b-9d6b-8b4d1afb58f5-kube-api-access-q9kmm\") pod \"maas-api-revoke-sub-746ab303-8qpf8\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:25.110297 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:25.110218 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:25.209765 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:25.209730 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f" path="/var/lib/kubelet/pods/58b59d3b-6e05-4bd9-af71-7bf47cd6bc7f/volumes" Nov 03 08:56:25.488306 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:25.488278 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8"] Nov 03 08:56:25.489545 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:56:25.489516 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5552d74_e374_456b_9d6b_8b4d1afb58f5.slice/crio-7b138a2da866550177cd581c9b04e61727ccc4a31fdbd9564b35becf0cd6ef38 WatchSource:0}: Error finding container 7b138a2da866550177cd581c9b04e61727ccc4a31fdbd9564b35becf0cd6ef38: Status 404 returned error can't find the container with id 7b138a2da866550177cd581c9b04e61727ccc4a31fdbd9564b35becf0cd6ef38 Nov 03 08:56:25.743603 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:25.743570 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" event={"ID":"a5552d74-e374-456b-9d6b-8b4d1afb58f5","Type":"ContainerStarted","Data":"7b138a2da866550177cd581c9b04e61727ccc4a31fdbd9564b35becf0cd6ef38"} Nov 03 08:56:26.751679 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:26.751641 5015 generic.go:356] "Generic (PLEG): container finished" podID="a5552d74-e374-456b-9d6b-8b4d1afb58f5" containerID="0e97df507cb51a34dbafd624d5f1c1e6774362a838bc7d4221dd883183b6b1ab" exitCode=0 Nov 03 08:56:26.752091 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:26.751731 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" event={"ID":"a5552d74-e374-456b-9d6b-8b4d1afb58f5","Type":"ContainerDied","Data":"0e97df507cb51a34dbafd624d5f1c1e6774362a838bc7d4221dd883183b6b1ab"} Nov 03 08:56:27.912456 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:27.912431 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:27.952685 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:27.952654 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-xks-ca\") pod \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " Nov 03 08:56:27.952832 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:27.952795 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5552d74-e374-456b-9d6b-8b4d1afb58f5-kube-api-access-q9kmm\" (UniqueName: \"kubernetes.io/projected/a5552d74-e374-456b-9d6b-8b4d1afb58f5-kube-api-access-q9kmm\") pod \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " Nov 03 08:56:27.952892 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:27.952868 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-openshift-ca\") pod \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\" (UID: \"a5552d74-e374-456b-9d6b-8b4d1afb58f5\") " Nov 03 08:56:27.953165 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:27.953141 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-openshift-ca" pod "a5552d74-e374-456b-9d6b-8b4d1afb58f5" (UID: "a5552d74-e374-456b-9d6b-8b4d1afb58f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:56:27.954996 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:27.954971 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5552d74-e374-456b-9d6b-8b4d1afb58f5-kube-api-access-q9kmm" pod "a5552d74-e374-456b-9d6b-8b4d1afb58f5" (UID: "a5552d74-e374-456b-9d6b-8b4d1afb58f5"). InnerVolumeSpecName "kube-api-access-q9kmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:56:27.955180 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:27.955158 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-xks-ca" pod "a5552d74-e374-456b-9d6b-8b4d1afb58f5" (UID: "a5552d74-e374-456b-9d6b-8b4d1afb58f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:56:28.054435 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:28.054367 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9kmm\" (UniqueName: \"kubernetes.io/projected/a5552d74-e374-456b-9d6b-8b4d1afb58f5-kube-api-access-q9kmm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:28.054435 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:28.054394 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:28.054435 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:28.054404 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5552d74-e374-456b-9d6b-8b4d1afb58f5-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:28.766438 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:28.766405 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" event={"ID":"a5552d74-e374-456b-9d6b-8b4d1afb58f5","Type":"ContainerDied","Data":"7b138a2da866550177cd581c9b04e61727ccc4a31fdbd9564b35becf0cd6ef38"} Nov 03 08:56:28.766655 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:28.766455 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b138a2da866550177cd581c9b04e61727ccc4a31fdbd9564b35becf0cd6ef38" Nov 03 08:56:28.766655 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:28.766421 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8" Nov 03 08:56:44.455564 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.455465 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv"] Nov 03 08:56:44.456197 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.456175 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a5552d74-e374-456b-9d6b-8b4d1afb58f5" containerName="revoke-keys" Nov 03 08:56:44.456292 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.456199 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5552d74-e374-456b-9d6b-8b4d1afb58f5" containerName="revoke-keys" Nov 03 08:56:44.456372 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.456359 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5552d74-e374-456b-9d6b-8b4d1afb58f5" containerName="revoke-keys" Nov 03 08:56:44.459698 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.459673 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.538277 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.538247 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170b3676-771c-4739-b132-380b3b25972a-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.538277 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.538281 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mklq\" (UniqueName: \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-kube-api-access-9mklq\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.538565 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.538311 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.538565 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.538406 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.538565 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.538462 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170b3676-771c-4739-b132-380b3b25972a-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.538565 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.538521 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.538717 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.538593 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.538717 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.538622 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.538717 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.538646 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.565852 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.565825 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-910d50-openshift-default-dockercfg-w8xwd\"" Nov 03 08:56:44.568743 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.568725 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv"] Nov 03 08:56:44.640172 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640136 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640350 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640204 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170b3676-771c-4739-b132-380b3b25972a-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640350 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640231 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mklq\" (UniqueName: \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-kube-api-access-9mklq\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640350 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640271 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640350 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640314 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640619 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640382 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170b3676-771c-4739-b132-380b3b25972a-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640619 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640413 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640619 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640469 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640619 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640546 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640810 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640636 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640810 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640751 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640810 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640792 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.640983 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.640962 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.641069 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.641053 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170b3676-771c-4739-b132-380b3b25972a-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.643059 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.643040 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.643362 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.643342 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170b3676-771c-4739-b132-380b3b25972a-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.700087 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.700055 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.700321 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.700304 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mklq\" (UniqueName: \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-kube-api-access-9mklq\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:44.773465 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:44.773426 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:45.207541 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:45.207514 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv"] Nov 03 08:56:45.208128 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:56:45.208106 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170b3676_771c_4739_b132_380b3b25972a.slice/crio-3fd98063e624af0f34ba80274f6bef52849214663f6cef5c56f94f70ece2c797 WatchSource:0}: Error finding container 3fd98063e624af0f34ba80274f6bef52849214663f6cef5c56f94f70ece2c797: Status 404 returned error can't find the container with id 3fd98063e624af0f34ba80274f6bef52849214663f6cef5c56f94f70ece2c797 Nov 03 08:56:45.210502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:45.210447 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:56:45.210588 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:45.210523 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:56:45.210588 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:45.210574 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:56:45.886135 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:45.886096 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" event={"ID":"170b3676-771c-4739-b132-380b3b25972a","Type":"ContainerStarted","Data":"c1d0d720e0790e9e98bc5a3df2119edef04ca9e6f5b600186623279f8a64ad90"} Nov 03 08:56:45.886670 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:45.886149 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" event={"ID":"170b3676-771c-4739-b132-380b3b25972a","Type":"ContainerStarted","Data":"3fd98063e624af0f34ba80274f6bef52849214663f6cef5c56f94f70ece2c797"} Nov 03 08:56:46.077216 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:46.077138 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" podStartSLOduration=2.07711518 podStartE2EDuration="2.07711518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:56:46.06081993 +0000 UTC m=+1038.985762161" watchObservedRunningTime="2026-11-03 08:56:46.07711518 +0000 UTC m=+1039.002057397" Nov 03 08:56:46.774140 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:46.774093 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:46.776127 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:46.776097 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:56:46.776290 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:46.776155 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" podUID="170b3676-771c-4739-b132-380b3b25972a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 08:56:46.902655 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:56:46.902620 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:56:47.776889 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:47.776851 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:56:47.777047 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:47.776910 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" podUID="170b3676-771c-4739-b132-380b3b25972a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 08:56:48.277224 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.277185 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr"] Nov 03 08:56:48.288539 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.288508 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.377511 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.376769 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr"] Nov 03 08:56:48.396808 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.396769 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 08:56:48.396808 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.396780 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 08:56:48.397117 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.396902 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 08:56:48.397117 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.396952 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 08:56:48.401367 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.401342 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-provider-credentials\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.401492 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.401396 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-routing-overlay-volume\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.401578 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.401565 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtjq\" (UniqueName: \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-kube-api-access-zjtjq\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.401621 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.401589 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-plugins-config-volume\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.415080 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.415058 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5x2kk\"" Nov 03 08:56:48.415080 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.415061 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c3fc4b163c39\"" Nov 03 08:56:48.415221 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.415191 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7c1592aed25b\"" Nov 03 08:56:48.503999 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.503968 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-provider-credentials\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.504202 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.504190 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-routing-overlay-volume\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.504422 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.504409 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtjq\" (UniqueName: \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-kube-api-access-zjtjq\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.504542 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.504527 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-plugins-config-volume\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.504918 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.504868 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-routing-overlay-volume\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.505391 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.505365 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-plugins-config-volume\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.507738 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.507715 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-provider-credentials\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.576825 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.576730 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtjq\" (UniqueName: \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-kube-api-access-zjtjq\") pod \"payload-processing-external-model-5cb89b647c-2zxdr\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.605334 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.605298 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:48.775532 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.774925 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:56:48.775532 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:48.774991 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" podUID="170b3676-771c-4739-b132-380b3b25972a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 08:56:49.031767 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:49.031330 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr"] Nov 03 08:56:49.034562 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:56:49.034528 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6184fbeb_5fe0_43b8_8749_844a994834a0.slice/crio-e22db8f515babab1d3d37d783f0bb4681807499ff74bd3e34cf356b0c34071f7 WatchSource:0}: Error finding container e22db8f515babab1d3d37d783f0bb4681807499ff74bd3e34cf356b0c34071f7: Status 404 returned error can't find the container with id e22db8f515babab1d3d37d783f0bb4681807499ff74bd3e34cf356b0c34071f7 Nov 03 08:56:49.773890 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:49.773851 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:56:49.774528 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:49.774465 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" podUID="170b3676-771c-4739-b132-380b3b25972a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 08:56:49.915521 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:49.915470 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" event={"ID":"6184fbeb-5fe0-43b8-8749-844a994834a0","Type":"ContainerStarted","Data":"e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad"} Nov 03 08:56:49.915827 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:49.915807 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" event={"ID":"6184fbeb-5fe0-43b8-8749-844a994834a0","Type":"ContainerStarted","Data":"e22db8f515babab1d3d37d783f0bb4681807499ff74bd3e34cf356b0c34071f7"} Nov 03 08:56:49.915979 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:49.915969 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:56:50.045563 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:50.039429 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" podStartSLOduration=2.039409819 podStartE2EDuration="2.039409819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:56:50.035954672 +0000 UTC m=+1042.960896938" watchObservedRunningTime="2026-11-03 08:56:50.039409819 +0000 UTC m=+1042.964352038" Nov 03 08:56:50.781639 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:50.781591 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:56:50.782242 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:50.782212 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" podUID="170b3676-771c-4739-b132-380b3b25972a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 08:56:51.774168 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:51.774075 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:56:51.774168 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:51.774161 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" podUID="170b3676-771c-4739-b132-380b3b25972a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 08:56:53.424928 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:53.424614 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:53.425418 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:53.425193 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:56:53.426889 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:56:53.426865 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 08:57:00.962750 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:00.962714 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:57:09.950368 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:09.950335 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f"] Nov 03 08:57:09.955766 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:09.955742 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.010462 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.010429 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-26e861-openshift-default-dockercfg-4wl7s\"" Nov 03 08:57:10.052264 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.052219 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.052631 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.052603 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.052775 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.052643 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.052775 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.052750 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7kgr\" (UniqueName: \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-kube-api-access-b7kgr\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.052933 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.052910 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.053034 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.053019 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.053095 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.053083 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.053147 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.053116 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.053242 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.053222 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.065377 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.065341 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f"] Nov 03 08:57:10.154408 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.154380 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.154636 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.154454 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.154636 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.154526 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.154636 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.154553 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.154636 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.154585 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.154636 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.154620 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.154874 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.154655 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.154874 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.154677 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.154874 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.154714 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7kgr\" (UniqueName: \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-kube-api-access-b7kgr\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.155048 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.155026 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.155111 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.155072 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.155168 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.155127 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.155236 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.155212 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.155316 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.155290 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.157329 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.157307 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.157652 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.157629 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.216032 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.215953 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.216032 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.215999 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7kgr\" (UniqueName: \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-kube-api-access-b7kgr\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.272783 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.272751 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:10.713401 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.713369 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f"] Nov 03 08:57:10.748783 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:57:10.748756 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762691b8_4d8b_41b4_b0fd_b1f64b7f6fe4.slice/crio-7025033a3839505a4e0ee39c3f38fc640a3097ad02eceec1af6e65a5dfab82a8 WatchSource:0}: Error finding container 7025033a3839505a4e0ee39c3f38fc640a3097ad02eceec1af6e65a5dfab82a8: Status 404 returned error can't find the container with id 7025033a3839505a4e0ee39c3f38fc640a3097ad02eceec1af6e65a5dfab82a8 Nov 03 08:57:10.751316 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.751272 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:57:10.751394 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.751370 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:57:10.751431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:10.751409 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:57:11.153927 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:11.153895 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" event={"ID":"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4","Type":"ContainerStarted","Data":"c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b"} Nov 03 08:57:11.154424 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:11.153954 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" event={"ID":"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4","Type":"ContainerStarted","Data":"7025033a3839505a4e0ee39c3f38fc640a3097ad02eceec1af6e65a5dfab82a8"} Nov 03 08:57:11.278998 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:11.278902 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" podStartSLOduration=2.278880135 podStartE2EDuration="2.278880135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:57:11.273962569 +0000 UTC m=+1064.198904788" watchObservedRunningTime="2026-11-03 08:57:11.278880135 +0000 UTC m=+1064.203822353" Nov 03 08:57:11.285237 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:11.285186 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:57:11.286296 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:11.281051 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:11.286451 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:11.286369 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 08:57:12.273491 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:12.273425 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:57:12.273996 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:12.273530 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 08:57:13.273969 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:13.273930 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:57:13.274369 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:13.274008 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 08:57:14.273357 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:14.273321 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:57:14.273569 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:14.273384 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 08:57:15.273993 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:15.273952 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:57:15.274551 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:15.274015 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 08:57:16.274117 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:16.274083 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:57:16.274562 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:16.274136 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 08:57:16.903492 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:57:16.903440 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:57:17.472672 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:17.472640 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:17.473242 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:17.473218 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:17.474185 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:17.474162 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 08:57:18.683035 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.683000 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw"] Nov 03 08:57:18.687216 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.687199 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.797408 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.797377 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.797606 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.797429 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.797653 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.797592 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d138326-bc46-4f60-9606-caa8d164b7cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.797688 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.797667 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.797759 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.797740 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.797798 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.797779 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.797853 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.797839 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.797887 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.797866 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.797917 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.797889 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nkhr\" (UniqueName: \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-kube-api-access-7nkhr\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.899189 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.899150 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.899189 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.899194 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.899437 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.899251 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.899437 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.899272 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.899437 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.899297 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nkhr\" (UniqueName: \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-kube-api-access-7nkhr\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.899437 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.899358 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.899437 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.899405 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.899729 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.899495 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d138326-bc46-4f60-9606-caa8d164b7cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.899729 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.899561 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.899729 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.899649 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.900087 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.900061 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.900427 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.900404 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.900642 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.900458 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.900642 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.900542 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d138326-bc46-4f60-9606-caa8d164b7cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.902355 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.902329 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.902464 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.902371 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.905855 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.905831 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw"] Nov 03 08:57:18.958985 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.958915 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nkhr\" (UniqueName: \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-kube-api-access-7nkhr\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:18.959174 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:18.959151 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z85qw\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:19.002142 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:19.002111 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:19.433387 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:19.433357 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw"] Nov 03 08:57:19.434185 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:57:19.434159 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d138326_bc46_4f60_9606_caa8d164b7cc.slice/crio-c992567ffed35e29dec03cda1daa63d2e4f4b406a8e98ea2bd55e0f76a8e05f4 WatchSource:0}: Error finding container c992567ffed35e29dec03cda1daa63d2e4f4b406a8e98ea2bd55e0f76a8e05f4: Status 404 returned error can't find the container with id c992567ffed35e29dec03cda1daa63d2e4f4b406a8e98ea2bd55e0f76a8e05f4 Nov 03 08:57:19.436714 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:19.436677 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:57:19.436782 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:19.436748 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:57:19.436829 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:19.436787 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:57:20.243511 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:20.243040 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" event={"ID":"6d138326-bc46-4f60-9606-caa8d164b7cc","Type":"ContainerStarted","Data":"c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46"} Nov 03 08:57:20.243511 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:20.243110 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" event={"ID":"6d138326-bc46-4f60-9606-caa8d164b7cc","Type":"ContainerStarted","Data":"c992567ffed35e29dec03cda1daa63d2e4f4b406a8e98ea2bd55e0f76a8e05f4"} Nov 03 08:57:20.374534 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:20.371339 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" podStartSLOduration=2.371304713 podStartE2EDuration="2.371304713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:57:20.367701673 +0000 UTC m=+1073.292643906" watchObservedRunningTime="2026-11-03 08:57:20.371304713 +0000 UTC m=+1073.296246931" Nov 03 08:57:21.004502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.003800 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:21.007140 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.005509 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z85qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:57:21.007140 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.005566 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 08:57:21.130465 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.130433 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm"] Nov 03 08:57:21.134744 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.134728 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.164491 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.164435 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-provider-credentials\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.164620 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.164582 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-plugins-config-volume\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.164688 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.164666 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-routing-overlay-volume\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.164832 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.164812 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2shqn\" (UniqueName: \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-kube-api-access-2shqn\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.192678 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.192650 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm"] Nov 03 08:57:21.266143 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.266043 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-plugins-config-volume\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.266143 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.266094 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-routing-overlay-volume\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.266630 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.266282 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2shqn\" (UniqueName: \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-kube-api-access-2shqn\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.266630 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.266343 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-provider-credentials\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.266891 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.266862 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-plugins-config-volume\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.266997 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.266894 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-routing-overlay-volume\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.269955 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.269933 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-provider-credentials\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.327454 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.327421 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2shqn\" (UniqueName: \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-kube-api-access-2shqn\") pod \"payload-processing-external-model-7dc8d75b5c-9jbxm\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.473406 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.473372 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:21.865620 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:57:21.865585 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2b7588_3f88_45c2_87c3_64c7d858fa0a.slice/crio-536203a1cf45ab7cd6887d1ea8e79a38d8f6261282cb33bbf82baf5f321a6b12 WatchSource:0}: Error finding container 536203a1cf45ab7cd6887d1ea8e79a38d8f6261282cb33bbf82baf5f321a6b12: Status 404 returned error can't find the container with id 536203a1cf45ab7cd6887d1ea8e79a38d8f6261282cb33bbf82baf5f321a6b12 Nov 03 08:57:21.865965 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:21.865945 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm"] Nov 03 08:57:22.002596 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:22.002562 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z85qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:57:22.002810 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:22.002631 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 08:57:22.282277 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:22.282232 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" event={"ID":"df2b7588-3f88-45c2-87c3-64c7d858fa0a","Type":"ContainerStarted","Data":"d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd"} Nov 03 08:57:22.282742 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:22.282288 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" event={"ID":"df2b7588-3f88-45c2-87c3-64c7d858fa0a","Type":"ContainerStarted","Data":"536203a1cf45ab7cd6887d1ea8e79a38d8f6261282cb33bbf82baf5f321a6b12"} Nov 03 08:57:22.402578 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:22.402522 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" podStartSLOduration=1.402499895 podStartE2EDuration="1.402499895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:57:22.400848563 +0000 UTC m=+1075.325790781" watchObservedRunningTime="2026-11-03 08:57:22.402499895 +0000 UTC m=+1075.327442112" Nov 03 08:57:23.003329 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:23.003293 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z85qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:57:23.003536 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:23.003353 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 08:57:23.289401 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:23.289326 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:24.004546 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:24.004469 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z85qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:57:24.004879 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:24.004841 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 08:57:25.002565 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:25.002528 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z85qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:57:25.002952 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:25.002584 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 08:57:26.002943 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:26.002907 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z85qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:57:26.003368 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:26.002961 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 08:57:27.002895 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:27.002857 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z85qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:57:27.003109 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:27.002913 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 08:57:28.479194 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:28.479162 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:28.480338 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:28.480317 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:28.482169 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:28.482155 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 08:57:34.299739 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:34.299700 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" podUID="df2b7588-3f88-45c2-87c3-64c7d858fa0a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:57:42.208497 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.208439 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm"] Nov 03 08:57:42.209911 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.209879 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" podUID="df2b7588-3f88-45c2-87c3-64c7d858fa0a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:57:42.242677 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.242645 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n"] Nov 03 08:57:42.246558 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.246538 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.323185 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.323148 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n"] Nov 03 08:57:42.383830 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.383797 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-plugins-config-volume\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.383830 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.383832 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-routing-overlay-volume\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.384061 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.384007 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-provider-credentials\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.384158 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.384141 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw7gf\" (UniqueName: \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-kube-api-access-kw7gf\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.485021 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.484980 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-plugins-config-volume\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.485021 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.485015 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-routing-overlay-volume\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.485299 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.485081 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-provider-credentials\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.485299 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.485166 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw7gf\" (UniqueName: \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-kube-api-access-kw7gf\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.485825 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.485792 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-routing-overlay-volume\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.486123 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.485863 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-plugins-config-volume\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.487874 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.487847 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-provider-credentials\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.545430 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.545388 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw7gf\" (UniqueName: \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-kube-api-access-kw7gf\") pod \"payload-processing-external-model-6845cdf9f-6l85n\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.561696 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.561665 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:42.946543 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:57:42.946496 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e2aa4c_c4ea_4451_b2f3_2b67c594959c.slice/crio-5942ce12373681dc39cc43f375e733b3198d65c40f485f00f587e4d28c32acf9 WatchSource:0}: Error finding container 5942ce12373681dc39cc43f375e733b3198d65c40f485f00f587e4d28c32acf9: Status 404 returned error can't find the container with id 5942ce12373681dc39cc43f375e733b3198d65c40f485f00f587e4d28c32acf9 Nov 03 08:57:42.948080 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:42.948046 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n"] Nov 03 08:57:43.521735 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:43.521684 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" event={"ID":"87e2aa4c-c4ea-4451-b2f3-2b67c594959c","Type":"ContainerStarted","Data":"882c7c734e571e3cd910e173692b0ff5dbfe517febd5166a52217f160a5e56d9"} Nov 03 08:57:43.522124 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:43.521753 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" event={"ID":"87e2aa4c-c4ea-4451-b2f3-2b67c594959c","Type":"ContainerStarted","Data":"5942ce12373681dc39cc43f375e733b3198d65c40f485f00f587e4d28c32acf9"} Nov 03 08:57:43.522124 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:43.521781 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:43.648799 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:43.648730 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" podStartSLOduration=1.648707054 podStartE2EDuration="1.648707054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:57:43.644894314 +0000 UTC m=+1096.569836549" watchObservedRunningTime="2026-11-03 08:57:43.648707054 +0000 UTC m=+1096.573649273" Nov 03 08:57:46.950829 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:57:46.950754 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:57:47.209029 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.208810 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" podUID="df2b7588-3f88-45c2-87c3-64c7d858fa0a" containerName="payload-processing" containerID="cri-o://d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd" gracePeriod=25 Nov 03 08:57:47.492421 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.492397 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:47.606440 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.606412 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-routing-overlay-volume\") pod \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " Nov 03 08:57:47.606679 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.606490 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-kube-api-access-2shqn\" (UniqueName: \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-kube-api-access-2shqn\") pod \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " Nov 03 08:57:47.606679 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.606597 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-provider-credentials\") pod \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " Nov 03 08:57:47.606679 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.606657 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-plugins-config-volume\") pod \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\" (UID: \"df2b7588-3f88-45c2-87c3-64c7d858fa0a\") " Nov 03 08:57:47.606949 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.606851 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-routing-overlay-volume" pod "df2b7588-3f88-45c2-87c3-64c7d858fa0a" (UID: "df2b7588-3f88-45c2-87c3-64c7d858fa0a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:57:47.607103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.607055 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-plugins-config-volume" pod "df2b7588-3f88-45c2-87c3-64c7d858fa0a" (UID: "df2b7588-3f88-45c2-87c3-64c7d858fa0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:57:47.607103 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.607100 5015 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-routing-overlay-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:57:47.609194 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.609170 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-provider-credentials" pod "df2b7588-3f88-45c2-87c3-64c7d858fa0a" (UID: "df2b7588-3f88-45c2-87c3-64c7d858fa0a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:57:47.609627 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.609595 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-kube-api-access-2shqn" pod "df2b7588-3f88-45c2-87c3-64c7d858fa0a" (UID: "df2b7588-3f88-45c2-87c3-64c7d858fa0a"). InnerVolumeSpecName "kube-api-access-2shqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:57:47.628371 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.628345 5015 generic.go:356] "Generic (PLEG): container finished" podID="df2b7588-3f88-45c2-87c3-64c7d858fa0a" containerID="d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd" exitCode=0 Nov 03 08:57:47.628501 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.628410 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" Nov 03 08:57:47.628501 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.628428 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" event={"ID":"df2b7588-3f88-45c2-87c3-64c7d858fa0a","Type":"ContainerDied","Data":"d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd"} Nov 03 08:57:47.628589 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.628519 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm" event={"ID":"df2b7588-3f88-45c2-87c3-64c7d858fa0a","Type":"ContainerDied","Data":"536203a1cf45ab7cd6887d1ea8e79a38d8f6261282cb33bbf82baf5f321a6b12"} Nov 03 08:57:47.628589 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.628547 5015 scope.go:122] "RemoveContainer" containerID="d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd" Nov 03 08:57:47.641955 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.641924 5015 scope.go:122] "RemoveContainer" containerID="d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd" Nov 03 08:57:47.642334 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:57:47.642304 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd\": container with ID starting with d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd not found: ID does not exist" containerID="d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd" Nov 03 08:57:47.642425 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.642346 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd"} err="failed to get container status \"d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd\": rpc error: code = NotFound desc = could not find container \"d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd\": container with ID starting with d1274d931964d79b431aa7d056994ca65eff8d471556e9b71210c69c02c8dfcd not found: ID does not exist" Nov 03 08:57:47.707812 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.707780 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2shqn\" (UniqueName: \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-kube-api-access-2shqn\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:57:47.707812 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.707809 5015 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df2b7588-3f88-45c2-87c3-64c7d858fa0a-provider-credentials\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:57:47.708001 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.707825 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2b7588-3f88-45c2-87c3-64c7d858fa0a-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:57:47.799662 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.799622 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm"] Nov 03 08:57:47.805210 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:47.805184 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dc8d75b5c-9jbxm"] Nov 03 08:57:49.204730 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:49.204699 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2b7588-3f88-45c2-87c3-64c7d858fa0a" path="/var/lib/kubelet/pods/df2b7588-3f88-45c2-87c3-64c7d858fa0a/volumes" Nov 03 08:57:50.478393 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.478362 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n"] Nov 03 08:57:50.481804 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.481767 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" podUID="87e2aa4c-c4ea-4451-b2f3-2b67c594959c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:57:50.505433 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.505408 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf"] Nov 03 08:57:50.506001 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.505987 5015 cpu_manager.go:414] "containerMap: removing container" podUID="df2b7588-3f88-45c2-87c3-64c7d858fa0a" containerName="payload-processing" Nov 03 08:57:50.506043 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.506003 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2b7588-3f88-45c2-87c3-64c7d858fa0a" containerName="payload-processing" Nov 03 08:57:50.506126 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.506117 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2b7588-3f88-45c2-87c3-64c7d858fa0a" containerName="payload-processing" Nov 03 08:57:50.510751 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.510737 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.565285 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.565255 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9ae5077d1953\"" Nov 03 08:57:50.565285 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.565259 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-205bf81aeb7a\"" Nov 03 08:57:50.590778 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.590748 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf"] Nov 03 08:57:50.642135 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.642105 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-provider-credentials\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.642298 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.642147 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-routing-overlay-volume\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.642298 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.642182 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-plugins-config-volume\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.642376 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.642335 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf9wz\" (UniqueName: \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-kube-api-access-qf9wz\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.743192 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.743116 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-plugins-config-volume\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.743336 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.743218 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9wz\" (UniqueName: \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-kube-api-access-qf9wz\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.743376 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.743355 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-provider-credentials\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.743425 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.743394 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-routing-overlay-volume\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.744120 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.744097 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-plugins-config-volume\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.744187 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.744097 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-routing-overlay-volume\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.746022 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.746002 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-provider-credentials\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.803878 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.803847 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9wz\" (UniqueName: \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-kube-api-access-qf9wz\") pod \"payload-processing-external-model-665b9fd955-8zqxf\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:50.824642 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:50.824615 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:51.205329 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:51.205301 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf"] Nov 03 08:57:51.206149 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:57:51.206125 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bf6832_6716_4958_82f5_fe4c3340090b.slice/crio-7f13ee83a77709d5f9d3212e57d3f7822f15dcb845c30ce707327233f468698b WatchSource:0}: Error finding container 7f13ee83a77709d5f9d3212e57d3f7822f15dcb845c30ce707327233f468698b: Status 404 returned error can't find the container with id 7f13ee83a77709d5f9d3212e57d3f7822f15dcb845c30ce707327233f468698b Nov 03 08:57:51.658427 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:51.658388 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" event={"ID":"c4bf6832-6716-4958-82f5-fe4c3340090b","Type":"ContainerStarted","Data":"4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c"} Nov 03 08:57:51.658885 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:51.658497 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" event={"ID":"c4bf6832-6716-4958-82f5-fe4c3340090b","Type":"ContainerStarted","Data":"7f13ee83a77709d5f9d3212e57d3f7822f15dcb845c30ce707327233f468698b"} Nov 03 08:57:51.658885 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:51.658526 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:57:51.777126 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:51.777077 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" podStartSLOduration=1.777063335 podStartE2EDuration="1.777063335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:57:51.775119994 +0000 UTC m=+1104.700062213" watchObservedRunningTime="2026-11-03 08:57:51.777063335 +0000 UTC m=+1104.702005551" Nov 03 08:57:55.479260 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.479191 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" podUID="87e2aa4c-c4ea-4451-b2f3-2b67c594959c" containerName="payload-processing" containerID="cri-o://882c7c734e571e3cd910e173692b0ff5dbfe517febd5166a52217f160a5e56d9" gracePeriod=25 Nov 03 08:57:55.693894 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.693861 5015 generic.go:356] "Generic (PLEG): container finished" podID="87e2aa4c-c4ea-4451-b2f3-2b67c594959c" containerID="882c7c734e571e3cd910e173692b0ff5dbfe517febd5166a52217f160a5e56d9" exitCode=0 Nov 03 08:57:55.694081 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.693913 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" event={"ID":"87e2aa4c-c4ea-4451-b2f3-2b67c594959c","Type":"ContainerDied","Data":"882c7c734e571e3cd910e173692b0ff5dbfe517febd5166a52217f160a5e56d9"} Nov 03 08:57:55.753656 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.753627 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:55.803757 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.803732 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-kube-api-access-kw7gf\" (UniqueName: \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-kube-api-access-kw7gf\") pod \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " Nov 03 08:57:55.803949 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.803787 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-routing-overlay-volume\") pod \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " Nov 03 08:57:55.803949 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.803857 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-provider-credentials\") pod \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " Nov 03 08:57:55.803949 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.803945 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-plugins-config-volume\") pod \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\" (UID: \"87e2aa4c-c4ea-4451-b2f3-2b67c594959c\") " Nov 03 08:57:55.804266 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.804234 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-routing-overlay-volume" pod "87e2aa4c-c4ea-4451-b2f3-2b67c594959c" (UID: "87e2aa4c-c4ea-4451-b2f3-2b67c594959c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:57:55.804416 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.804361 5015 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-routing-overlay-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:57:55.804416 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.804369 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-plugins-config-volume" pod "87e2aa4c-c4ea-4451-b2f3-2b67c594959c" (UID: "87e2aa4c-c4ea-4451-b2f3-2b67c594959c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:57:55.806149 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.806126 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-kube-api-access-kw7gf" pod "87e2aa4c-c4ea-4451-b2f3-2b67c594959c" (UID: "87e2aa4c-c4ea-4451-b2f3-2b67c594959c"). InnerVolumeSpecName "kube-api-access-kw7gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:57:55.806149 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.806138 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-provider-credentials" pod "87e2aa4c-c4ea-4451-b2f3-2b67c594959c" (UID: "87e2aa4c-c4ea-4451-b2f3-2b67c594959c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:57:55.905427 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.905393 5015 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-provider-credentials\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:57:55.905427 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.905421 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:57:55.905427 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:55.905433 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw7gf\" (UniqueName: \"kubernetes.io/projected/87e2aa4c-c4ea-4451-b2f3-2b67c594959c-kube-api-access-kw7gf\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:57:56.701639 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:56.701608 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" Nov 03 08:57:56.701639 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:56.701626 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n" event={"ID":"87e2aa4c-c4ea-4451-b2f3-2b67c594959c","Type":"ContainerDied","Data":"5942ce12373681dc39cc43f375e733b3198d65c40f485f00f587e4d28c32acf9"} Nov 03 08:57:56.702090 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:56.701714 5015 scope.go:122] "RemoveContainer" containerID="882c7c734e571e3cd910e173692b0ff5dbfe517febd5166a52217f160a5e56d9" Nov 03 08:57:56.871998 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:56.871966 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n"] Nov 03 08:57:56.929221 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:56.928978 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6l85n"] Nov 03 08:57:57.207028 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:57.206989 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e2aa4c-c4ea-4451-b2f3-2b67c594959c" path="/var/lib/kubelet/pods/87e2aa4c-c4ea-4451-b2f3-2b67c594959c/volumes" Nov 03 08:57:59.529687 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:59.529652 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z85qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:57:59.530340 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:57:59.530292 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:58:02.668149 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:02.668119 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 08:58:02.840336 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:02.840303 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr"] Nov 03 08:58:07.841734 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:07.841614 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" podUID="6184fbeb-5fe0-43b8-8749-844a994834a0" containerName="payload-processing" containerID="cri-o://e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad" gracePeriod=25 Nov 03 08:58:08.145694 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.145663 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:58:08.304682 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.304636 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-kube-api-access-zjtjq\" (UniqueName: \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-kube-api-access-zjtjq\") pod \"6184fbeb-5fe0-43b8-8749-844a994834a0\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " Nov 03 08:58:08.304887 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.304728 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-provider-credentials\") pod \"6184fbeb-5fe0-43b8-8749-844a994834a0\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " Nov 03 08:58:08.304887 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.304844 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-plugins-config-volume\") pod \"6184fbeb-5fe0-43b8-8749-844a994834a0\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " Nov 03 08:58:08.305031 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.304951 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-routing-overlay-volume\") pod \"6184fbeb-5fe0-43b8-8749-844a994834a0\" (UID: \"6184fbeb-5fe0-43b8-8749-844a994834a0\") " Nov 03 08:58:08.305837 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.305809 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-routing-overlay-volume" pod "6184fbeb-5fe0-43b8-8749-844a994834a0" (UID: "6184fbeb-5fe0-43b8-8749-844a994834a0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:58:08.306007 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.305824 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-plugins-config-volume" pod "6184fbeb-5fe0-43b8-8749-844a994834a0" (UID: "6184fbeb-5fe0-43b8-8749-844a994834a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:58:08.307726 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.307697 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-provider-credentials" pod "6184fbeb-5fe0-43b8-8749-844a994834a0" (UID: "6184fbeb-5fe0-43b8-8749-844a994834a0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:58:08.316055 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.316026 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-kube-api-access-zjtjq" pod "6184fbeb-5fe0-43b8-8749-844a994834a0" (UID: "6184fbeb-5fe0-43b8-8749-844a994834a0"). InnerVolumeSpecName "kube-api-access-zjtjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:58:08.406582 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.406448 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjtjq\" (UniqueName: \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-kube-api-access-zjtjq\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:58:08.406582 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.406513 5015 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6184fbeb-5fe0-43b8-8749-844a994834a0-provider-credentials\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:58:08.406582 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.406531 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:58:08.406582 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.406548 5015 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6184fbeb-5fe0-43b8-8749-844a994834a0-routing-overlay-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:58:08.810939 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.810902 5015 generic.go:356] "Generic (PLEG): container finished" podID="6184fbeb-5fe0-43b8-8749-844a994834a0" containerID="e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad" exitCode=0 Nov 03 08:58:08.811190 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.811167 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" Nov 03 08:58:08.811345 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.811071 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" event={"ID":"6184fbeb-5fe0-43b8-8749-844a994834a0","Type":"ContainerDied","Data":"e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad"} Nov 03 08:58:08.811510 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.811468 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr" event={"ID":"6184fbeb-5fe0-43b8-8749-844a994834a0","Type":"ContainerDied","Data":"e22db8f515babab1d3d37d783f0bb4681807499ff74bd3e34cf356b0c34071f7"} Nov 03 08:58:08.811667 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.811651 5015 scope.go:122] "RemoveContainer" containerID="e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad" Nov 03 08:58:08.827368 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.827349 5015 scope.go:122] "RemoveContainer" containerID="e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad" Nov 03 08:58:08.827671 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:58:08.827649 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad\": container with ID starting with e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad not found: ID does not exist" containerID="e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad" Nov 03 08:58:08.827762 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:08.827677 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad"} err="failed to get container status \"e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad\": rpc error: code = NotFound desc = could not find container \"e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad\": container with ID starting with e64e4cbcc8124c4a381bb9376b4863b809a8f62d242c61df697c5fe78d0b81ad not found: ID does not exist" Nov 03 08:58:09.013948 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:09.013716 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr"] Nov 03 08:58:09.093555 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:09.077531 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cb89b647c-2zxdr"] Nov 03 08:58:09.270308 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:09.270268 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6184fbeb-5fe0-43b8-8749-844a994834a0" path="/var/lib/kubelet/pods/6184fbeb-5fe0-43b8-8749-844a994834a0/volumes" Nov 03 08:58:10.675445 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.675413 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw"] Nov 03 08:58:10.676023 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.676006 5015 cpu_manager.go:414] "containerMap: removing container" podUID="87e2aa4c-c4ea-4451-b2f3-2b67c594959c" containerName="payload-processing" Nov 03 08:58:10.676067 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.676024 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e2aa4c-c4ea-4451-b2f3-2b67c594959c" containerName="payload-processing" Nov 03 08:58:10.676067 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.676053 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6184fbeb-5fe0-43b8-8749-844a994834a0" containerName="payload-processing" Nov 03 08:58:10.676067 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.676061 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6184fbeb-5fe0-43b8-8749-844a994834a0" containerName="payload-processing" Nov 03 08:58:10.676156 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.676147 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6184fbeb-5fe0-43b8-8749-844a994834a0" containerName="payload-processing" Nov 03 08:58:10.676187 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.676160 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e2aa4c-c4ea-4451-b2f3-2b67c594959c" containerName="payload-processing" Nov 03 08:58:10.680428 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.680408 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.740577 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.740542 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw"] Nov 03 08:58:10.785666 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.785628 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26e861\"/\"distinct-w1-26e861-kserve-self-signed-certs\"" Nov 03 08:58:10.838879 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.838837 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5kmw\" (UniqueName: \"kubernetes.io/projected/acffa72b-ec10-45dd-99fb-614a015f41ba-kube-api-access-p5kmw\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.839123 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.838903 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-tmp-dir\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.839123 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.839056 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-home\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.839227 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.839134 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-model-cache\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.839227 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.839168 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-dshm\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.839301 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.839239 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acffa72b-ec10-45dd-99fb-614a015f41ba-tls-certs\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.939980 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.939886 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acffa72b-ec10-45dd-99fb-614a015f41ba-tls-certs\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.939980 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.939960 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5kmw\" (UniqueName: \"kubernetes.io/projected/acffa72b-ec10-45dd-99fb-614a015f41ba-kube-api-access-p5kmw\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.940217 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.939989 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-tmp-dir\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.940217 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.940033 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-home\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.940217 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.940101 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-model-cache\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.940217 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.940130 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-dshm\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.940522 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.940448 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-home\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.940641 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.940489 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-model-cache\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.940641 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.940562 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-tmp-dir\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.942766 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.942740 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-dshm\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.942915 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.942896 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acffa72b-ec10-45dd-99fb-614a015f41ba-tls-certs\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:10.999988 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:10.999951 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5kmw\" (UniqueName: \"kubernetes.io/projected/acffa72b-ec10-45dd-99fb-614a015f41ba-kube-api-access-p5kmw\") pod \"distinct-w1-26e861-kserve-69899c5848-4phjw\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:11.296501 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:11.295697 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:11.684216 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:11.684185 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw"] Nov 03 08:58:11.684659 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:58:11.684411 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacffa72b_ec10_45dd_99fb_614a015f41ba.slice/crio-3b5cbfe3bf2e633c6f58b2f4d40853352cb79eddc625c03788efb4857d9023f8 WatchSource:0}: Error finding container 3b5cbfe3bf2e633c6f58b2f4d40853352cb79eddc625c03788efb4857d9023f8: Status 404 returned error can't find the container with id 3b5cbfe3bf2e633c6f58b2f4d40853352cb79eddc625c03788efb4857d9023f8 Nov 03 08:58:11.842859 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:11.842824 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" event={"ID":"acffa72b-ec10-45dd-99fb-614a015f41ba","Type":"ContainerStarted","Data":"e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce"} Nov 03 08:58:11.843028 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:11.842868 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" event={"ID":"acffa72b-ec10-45dd-99fb-614a015f41ba","Type":"ContainerStarted","Data":"3b5cbfe3bf2e633c6f58b2f4d40853352cb79eddc625c03788efb4857d9023f8"} Nov 03 08:58:11.963653 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:11.963548 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podStartSLOduration=1.963533509 podStartE2EDuration="1.963533509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:58:11.959717429 +0000 UTC m=+1124.884659649" watchObservedRunningTime="2026-11-03 08:58:11.963533509 +0000 UTC m=+1124.888475725" Nov 03 08:58:12.296459 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:12.296425 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:16.903415 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:58:16.903365 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:58:21.296659 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:21.296620 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:21.308939 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:21.308913 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:21.925552 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:58:21.925519 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 08:58:46.902128 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:58:46.902093 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:59:01.486227 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.486189 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4"] Nov 03 08:59:01.490018 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.490002 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.544662 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.544637 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-77207e-openshift-default-dockercfg-rcrfh\"" Nov 03 08:59:01.598545 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.598514 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4"] Nov 03 08:59:01.643147 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.643107 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbw6s\" (UniqueName: \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-kube-api-access-cbw6s\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.643323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.643225 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0627db0c-9fec-4819-b096-ed00eff6f193-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.643323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.643259 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.643323 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.643287 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.643446 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.643411 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.643509 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.643449 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0627db0c-9fec-4819-b096-ed00eff6f193-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.643547 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.643509 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.643581 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.643564 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.643640 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.643621 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.745250 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.745148 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.745431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.745268 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0627db0c-9fec-4819-b096-ed00eff6f193-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.745431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.745312 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.745431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.745380 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.745431 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.745422 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.745728 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.745572 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.745728 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.745582 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbw6s\" (UniqueName: \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-kube-api-access-cbw6s\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.745728 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.745655 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0627db0c-9fec-4819-b096-ed00eff6f193-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.745728 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.745693 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.745728 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.745724 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.746105 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.746087 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.746276 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.746247 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.746456 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.746399 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.746559 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.746539 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0627db0c-9fec-4819-b096-ed00eff6f193-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.748712 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.748688 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0627db0c-9fec-4819-b096-ed00eff6f193-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.748801 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.748712 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.804523 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.804490 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:01.804815 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:01.804795 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbw6s\" (UniqueName: \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-kube-api-access-cbw6s\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:02.104439 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:02.104356 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:02.540591 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:02.540555 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4"] Nov 03 08:59:02.541015 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:59:02.540847 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0627db0c_9fec_4819_b096_ed00eff6f193.slice/crio-3a9f68049451ed400bd447bfd5d3979459dcb87e2e6797b18fa31e5f74b0f951 WatchSource:0}: Error finding container 3a9f68049451ed400bd447bfd5d3979459dcb87e2e6797b18fa31e5f74b0f951: Status 404 returned error can't find the container with id 3a9f68049451ed400bd447bfd5d3979459dcb87e2e6797b18fa31e5f74b0f951 Nov 03 08:59:02.543380 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:02.543347 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:59:02.543514 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:02.543410 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:59:02.543514 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:02.543440 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 08:59:03.208993 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:03.208954 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" event={"ID":"0627db0c-9fec-4819-b096-ed00eff6f193","Type":"ContainerStarted","Data":"4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f"} Nov 03 08:59:03.209162 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:03.209015 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" event={"ID":"0627db0c-9fec-4819-b096-ed00eff6f193","Type":"ContainerStarted","Data":"3a9f68049451ed400bd447bfd5d3979459dcb87e2e6797b18fa31e5f74b0f951"} Nov 03 08:59:03.431211 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:03.431147 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" podStartSLOduration=2.431131081 podStartE2EDuration="2.431131081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:59:03.379569379 +0000 UTC m=+1176.304511593" watchObservedRunningTime="2026-11-03 08:59:03.431131081 +0000 UTC m=+1176.356073298" Nov 03 08:59:04.105382 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:04.105346 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:04.107106 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:04.107084 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 08:59:04.107197 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:04.107133 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 08:59:05.105299 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:05.105266 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 08:59:05.105502 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:05.105321 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 08:59:06.104663 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:06.104634 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 08:59:06.104837 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:06.104683 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 08:59:07.105351 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:07.105313 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 08:59:07.105789 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:07.105375 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 08:59:08.105303 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:08.105264 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 08:59:08.105634 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:08.105606 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 08:59:09.104949 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:09.104914 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 08:59:09.105150 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:09.104975 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 08:59:10.108879 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:10.108845 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:10.109327 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:10.109309 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:10.109915 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:10.109898 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 08:59:16.903578 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:59:16.903540 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:59:27.397342 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:27.397303 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 08:59:27.401973 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:27.401954 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 08:59:27.465256 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:27.465225 5015 scope.go:122] "RemoveContainer" containerID="241649e1f156b3d3653047ec4679a1d93280cfb2712413cdbf6c2558989f9951" Nov 03 08:59:46.902718 ip-10-0-131-14 kubenswrapper[5015]: E1103 08:59:46.902632 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 08:59:52.108213 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.108182 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r"] Nov 03 08:59:52.112308 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.112288 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.170198 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.170162 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r"] Nov 03 08:59:52.215092 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.215063 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"distinct-2-w2-77207e-kserve-self-signed-certs\"" Nov 03 08:59:52.232976 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.232940 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tls-certs\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.233125 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.233027 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-model-cache\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.233125 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.233066 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-home\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.233125 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.233111 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tmp-dir\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.233239 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.233174 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-dshm\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.233276 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.233235 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7s4q\" (UniqueName: \"kubernetes.io/projected/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-kube-api-access-f7s4q\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.334165 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.334124 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-model-cache\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.334376 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.334172 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-home\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.334376 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.334202 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tmp-dir\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.334376 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.334268 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-dshm\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.334376 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.334318 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7s4q\" (UniqueName: \"kubernetes.io/projected/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-kube-api-access-f7s4q\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.334632 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.334546 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tls-certs\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.334689 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.334643 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-model-cache\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.334689 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.334681 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tmp-dir\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.334782 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.334709 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-home\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.336888 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.336867 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-dshm\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.337293 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.337272 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tls-certs\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.394600 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.394516 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7s4q\" (UniqueName: \"kubernetes.io/projected/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-kube-api-access-f7s4q\") pod \"distinct-2-w2-77207e-kserve-6587649464-5s77r\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.425380 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.425348 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 08:59:52.845643 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:52.845611 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r"] Nov 03 08:59:52.846316 ip-10-0-131-14 kubenswrapper[5015]: W1103 08:59:52.846292 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56aad1c2_84bc_40fe_bba6_a0eb6374b47a.slice/crio-37f715123571e90555d8e1747f95c1db279d820a9998b4272546dab74718b842 WatchSource:0}: Error finding container 37f715123571e90555d8e1747f95c1db279d820a9998b4272546dab74718b842: Status 404 returned error can't find the container with id 37f715123571e90555d8e1747f95c1db279d820a9998b4272546dab74718b842 Nov 03 08:59:53.594795 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:53.594759 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" event={"ID":"56aad1c2-84bc-40fe-bba6-a0eb6374b47a","Type":"ContainerStarted","Data":"02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57"} Nov 03 08:59:53.595193 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:53.594804 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" event={"ID":"56aad1c2-84bc-40fe-bba6-a0eb6374b47a","Type":"ContainerStarted","Data":"37f715123571e90555d8e1747f95c1db279d820a9998b4272546dab74718b842"} Nov 03 08:59:53.713559 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:53.713505 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podStartSLOduration=1.713470333 podStartE2EDuration="1.713470333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:59:53.711434352 +0000 UTC m=+1226.636376586" watchObservedRunningTime="2026-11-03 08:59:53.713470333 +0000 UTC m=+1226.638412890" Nov 03 08:59:54.425993 ip-10-0-131-14 kubenswrapper[5015]: I1103 08:59:54.425962 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:00:00.182799 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.182757 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9"] Nov 03 09:00:00.187754 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.187730 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf"] Nov 03 09:00:00.187908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.187846 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" Nov 03 09:00:00.191929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.191910 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" Nov 03 09:00:00.205127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.205106 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4"] Nov 03 09:00:00.209087 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.209068 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" Nov 03 09:00:00.222988 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.222964 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894940-h7rmf\" (UID: \"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" Nov 03 09:00:00.223078 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.223014 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbd99\" (UniqueName: \"kubernetes.io/projected/43dcb009-df45-4a99-94fa-5e427b693335-kube-api-access-cbd99\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9\" (UID: \"43dcb009-df45-4a99-94fa-5e427b693335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" Nov 03 09:00:00.223225 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.223210 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43dcb009-df45-4a99-94fa-5e427b693335-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9\" (UID: \"43dcb009-df45-4a99-94fa-5e427b693335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" Nov 03 09:00:00.223266 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.223243 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbrlx\" (UniqueName: \"kubernetes.io/projected/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-kube-api-access-pbrlx\") pod \"maas-api-key-cleanup-29894940-h7rmf\" (UID: \"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" Nov 03 09:00:00.241210 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.241180 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:00:00.289551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.289522 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9"] Nov 03 09:00:00.293776 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.293733 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf"] Nov 03 09:00:00.324801 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.324767 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43dcb009-df45-4a99-94fa-5e427b693335-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9\" (UID: \"43dcb009-df45-4a99-94fa-5e427b693335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" Nov 03 09:00:00.324999 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.324827 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbrlx\" (UniqueName: \"kubernetes.io/projected/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-kube-api-access-pbrlx\") pod \"maas-api-key-cleanup-29894940-h7rmf\" (UID: \"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" Nov 03 09:00:00.324999 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.324896 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894940-h7rmf\" (UID: \"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" Nov 03 09:00:00.324999 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.324957 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbd99\" (UniqueName: \"kubernetes.io/projected/43dcb009-df45-4a99-94fa-5e427b693335-kube-api-access-cbd99\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9\" (UID: \"43dcb009-df45-4a99-94fa-5e427b693335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" Nov 03 09:00:00.324999 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.324992 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhntn\" (UniqueName: \"kubernetes.io/projected/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-kube-api-access-qhntn\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4\" (UID: \"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" Nov 03 09:00:00.325230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.325036 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4\" (UID: \"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" Nov 03 09:00:00.325996 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.325969 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894940-h7rmf\" (UID: \"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" Nov 03 09:00:00.326301 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.326275 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43dcb009-df45-4a99-94fa-5e427b693335-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9\" (UID: \"43dcb009-df45-4a99-94fa-5e427b693335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" Nov 03 09:00:00.338423 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.338395 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4"] Nov 03 09:00:00.386867 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.386768 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbd99\" (UniqueName: \"kubernetes.io/projected/43dcb009-df45-4a99-94fa-5e427b693335-kube-api-access-cbd99\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9\" (UID: \"43dcb009-df45-4a99-94fa-5e427b693335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" Nov 03 09:00:00.388597 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.388569 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbrlx\" (UniqueName: \"kubernetes.io/projected/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-kube-api-access-pbrlx\") pod \"maas-api-key-cleanup-29894940-h7rmf\" (UID: \"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" Nov 03 09:00:00.426796 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.426748 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhntn\" (UniqueName: \"kubernetes.io/projected/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-kube-api-access-qhntn\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4\" (UID: \"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" Nov 03 09:00:00.426976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.426811 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4\" (UID: \"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" Nov 03 09:00:00.427550 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.427527 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4\" (UID: \"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" Nov 03 09:00:00.487264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.487234 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhntn\" (UniqueName: \"kubernetes.io/projected/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-kube-api-access-qhntn\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4\" (UID: \"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" Nov 03 09:00:00.506799 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.506764 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" Nov 03 09:00:00.534566 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.515063 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" Nov 03 09:00:00.534566 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:00.528622 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" Nov 03 09:00:01.012434 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:01.012336 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf"] Nov 03 09:00:01.022152 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:00:01.022097 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d6cdf2_c15b_4be8_ba7a_90807a0a27a0.slice/crio-8b936c395ad42a380b503196b1b6f13970c18df093f6fbb1a261951554f664aa WatchSource:0}: Error finding container 8b936c395ad42a380b503196b1b6f13970c18df093f6fbb1a261951554f664aa: Status 404 returned error can't find the container with id 8b936c395ad42a380b503196b1b6f13970c18df093f6fbb1a261951554f664aa Nov 03 09:00:01.054507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:01.052874 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4"] Nov 03 09:00:01.057843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:01.057810 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9"] Nov 03 09:00:01.061911 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:00:01.061885 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43dcb009_df45_4a99_94fa_5e427b693335.slice/crio-e8b69dec8b46e7dc18a7eeaf9f7555d5fb027f02ce4ac1ee43ddcdd7cee39cd5 WatchSource:0}: Error finding container e8b69dec8b46e7dc18a7eeaf9f7555d5fb027f02ce4ac1ee43ddcdd7cee39cd5: Status 404 returned error can't find the container with id e8b69dec8b46e7dc18a7eeaf9f7555d5fb027f02ce4ac1ee43ddcdd7cee39cd5 Nov 03 09:00:01.705558 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:01.703417 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" event={"ID":"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac","Type":"ContainerStarted","Data":"063179100da30fcfc60afda585f98dba9b461ed1a03ef550fec9c344a20f217a"} Nov 03 09:00:01.705558 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:01.703507 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" event={"ID":"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac","Type":"ContainerStarted","Data":"e09d015eeb8ada132402653e73201b8634d1fec8727e1dcc337593f70d77f75d"} Nov 03 09:00:01.707607 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:01.706853 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" event={"ID":"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0","Type":"ContainerStarted","Data":"15f5d97b6113857920858d8690290c038500f30cb6d74b660262c62f07f6ec7a"} Nov 03 09:00:01.707607 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:01.706913 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" event={"ID":"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0","Type":"ContainerStarted","Data":"8b936c395ad42a380b503196b1b6f13970c18df093f6fbb1a261951554f664aa"} Nov 03 09:00:01.710747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:01.710035 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" event={"ID":"43dcb009-df45-4a99-94fa-5e427b693335","Type":"ContainerStarted","Data":"e9dc8bccdde0c1a863a9d9fd5e5c410996343357109d6ba333d86512cb94117b"} Nov 03 09:00:01.710747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:01.710095 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" event={"ID":"43dcb009-df45-4a99-94fa-5e427b693335","Type":"ContainerStarted","Data":"e8b69dec8b46e7dc18a7eeaf9f7555d5fb027f02ce4ac1ee43ddcdd7cee39cd5"} Nov 03 09:00:01.823631 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:01.823570 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" podStartSLOduration=1.823554396 podStartE2EDuration="1.823554396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:00:01.821022764 +0000 UTC m=+1234.745965026" watchObservedRunningTime="2026-11-03 09:00:01.823554396 +0000 UTC m=+1234.748496613" Nov 03 09:00:01.945501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:01.942145 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" podStartSLOduration=1.942123507 podStartE2EDuration="1.942123507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:00:01.940874878 +0000 UTC m=+1234.865817098" watchObservedRunningTime="2026-11-03 09:00:01.942123507 +0000 UTC m=+1234.867065732" Nov 03 09:00:02.077674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.077604 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" podStartSLOduration=2.077582813 podStartE2EDuration="2.077582813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:00:02.072381176 +0000 UTC m=+1234.997323418" watchObservedRunningTime="2026-11-03 09:00:02.077582813 +0000 UTC m=+1235.002525030" Nov 03 09:00:02.238740 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:02.238693 5015 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462c25e5_1a1c_4e7d_bc0a_c650e15f1dac.slice/crio-conmon-063179100da30fcfc60afda585f98dba9b461ed1a03ef550fec9c344a20f217a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 09:00:02.426295 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.426199 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:00:02.441880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.441845 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:00:02.719645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.719566 5015 generic.go:356] "Generic (PLEG): container finished" podID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerID="063179100da30fcfc60afda585f98dba9b461ed1a03ef550fec9c344a20f217a" exitCode=60 Nov 03 09:00:02.720204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.719717 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" event={"ID":"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac","Type":"ContainerDied","Data":"063179100da30fcfc60afda585f98dba9b461ed1a03ef550fec9c344a20f217a"} Nov 03 09:00:02.720204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.720017 5015 scope.go:122] "RemoveContainer" containerID="063179100da30fcfc60afda585f98dba9b461ed1a03ef550fec9c344a20f217a" Nov 03 09:00:02.721461 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.721427 5015 generic.go:356] "Generic (PLEG): container finished" podID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerID="15f5d97b6113857920858d8690290c038500f30cb6d74b660262c62f07f6ec7a" exitCode=60 Nov 03 09:00:02.721613 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.721506 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" event={"ID":"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0","Type":"ContainerDied","Data":"15f5d97b6113857920858d8690290c038500f30cb6d74b660262c62f07f6ec7a"} Nov 03 09:00:02.721804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.721786 5015 scope.go:122] "RemoveContainer" containerID="15f5d97b6113857920858d8690290c038500f30cb6d74b660262c62f07f6ec7a" Nov 03 09:00:02.723529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.723428 5015 generic.go:356] "Generic (PLEG): container finished" podID="43dcb009-df45-4a99-94fa-5e427b693335" containerID="e9dc8bccdde0c1a863a9d9fd5e5c410996343357109d6ba333d86512cb94117b" exitCode=60 Nov 03 09:00:02.723529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.723516 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" event={"ID":"43dcb009-df45-4a99-94fa-5e427b693335","Type":"ContainerDied","Data":"e9dc8bccdde0c1a863a9d9fd5e5c410996343357109d6ba333d86512cb94117b"} Nov 03 09:00:02.723759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.723745 5015 scope.go:122] "RemoveContainer" containerID="e9dc8bccdde0c1a863a9d9fd5e5c410996343357109d6ba333d86512cb94117b" Nov 03 09:00:02.739190 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:02.739157 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:00:03.733284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.733253 5015 generic.go:356] "Generic (PLEG): container finished" podID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerID="983704bbc6a9c78aefb8fa9c19140d0d12d8dd0ce9cde5e97e34499e480d6dff" exitCode=60 Nov 03 09:00:03.733733 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.733335 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" event={"ID":"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0","Type":"ContainerDied","Data":"983704bbc6a9c78aefb8fa9c19140d0d12d8dd0ce9cde5e97e34499e480d6dff"} Nov 03 09:00:03.733733 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.733402 5015 scope.go:122] "RemoveContainer" containerID="15f5d97b6113857920858d8690290c038500f30cb6d74b660262c62f07f6ec7a" Nov 03 09:00:03.733733 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.733705 5015 scope.go:122] "RemoveContainer" containerID="983704bbc6a9c78aefb8fa9c19140d0d12d8dd0ce9cde5e97e34499e480d6dff" Nov 03 09:00:03.734042 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:03.734018 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894940-h7rmf_odh-ai-gateway-infra(50d6cdf2-c15b-4be8-ba7a-90807a0a27a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" Nov 03 09:00:03.735734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.735491 5015 generic.go:356] "Generic (PLEG): container finished" podID="43dcb009-df45-4a99-94fa-5e427b693335" containerID="60bc2971ba7b7afb8e03084e3eb8d207fe4bf93804ac0e89cfe3714e11c332c9" exitCode=60 Nov 03 09:00:03.735734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.735573 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" event={"ID":"43dcb009-df45-4a99-94fa-5e427b693335","Type":"ContainerDied","Data":"60bc2971ba7b7afb8e03084e3eb8d207fe4bf93804ac0e89cfe3714e11c332c9"} Nov 03 09:00:03.735862 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.735851 5015 scope.go:122] "RemoveContainer" containerID="60bc2971ba7b7afb8e03084e3eb8d207fe4bf93804ac0e89cfe3714e11c332c9" Nov 03 09:00:03.736084 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:03.736062 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9_odh-ai-gateway-infra(43dcb009-df45-4a99-94fa-5e427b693335)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" podUID="43dcb009-df45-4a99-94fa-5e427b693335" Nov 03 09:00:03.737729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.737708 5015 generic.go:356] "Generic (PLEG): container finished" podID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerID="754cba415eb68b5c85e1c8d1d055f4a47d588f9575b117754980b0dcbc9743a5" exitCode=60 Nov 03 09:00:03.737820 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.737763 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" event={"ID":"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac","Type":"ContainerDied","Data":"754cba415eb68b5c85e1c8d1d055f4a47d588f9575b117754980b0dcbc9743a5"} Nov 03 09:00:03.738020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.738004 5015 scope.go:122] "RemoveContainer" containerID="754cba415eb68b5c85e1c8d1d055f4a47d588f9575b117754980b0dcbc9743a5" Nov 03 09:00:03.738222 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:03.738206 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4_odh-ai-gateway-infra(462c25e5-1a1c-4e7d-bc0a-c650e15f1dac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" Nov 03 09:00:03.831097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.831073 5015 scope.go:122] "RemoveContainer" containerID="e9dc8bccdde0c1a863a9d9fd5e5c410996343357109d6ba333d86512cb94117b" Nov 03 09:00:03.845748 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:03.845728 5015 scope.go:122] "RemoveContainer" containerID="063179100da30fcfc60afda585f98dba9b461ed1a03ef550fec9c344a20f217a" Nov 03 09:00:04.748876 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:04.748845 5015 scope.go:122] "RemoveContainer" containerID="754cba415eb68b5c85e1c8d1d055f4a47d588f9575b117754980b0dcbc9743a5" Nov 03 09:00:04.749370 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:04.749178 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4_odh-ai-gateway-infra(462c25e5-1a1c-4e7d-bc0a-c650e15f1dac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" Nov 03 09:00:04.750725 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:04.750702 5015 scope.go:122] "RemoveContainer" containerID="983704bbc6a9c78aefb8fa9c19140d0d12d8dd0ce9cde5e97e34499e480d6dff" Nov 03 09:00:04.750982 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:04.750955 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894940-h7rmf_odh-ai-gateway-infra(50d6cdf2-c15b-4be8-ba7a-90807a0a27a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" Nov 03 09:00:04.752550 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:04.752529 5015 scope.go:122] "RemoveContainer" containerID="60bc2971ba7b7afb8e03084e3eb8d207fe4bf93804ac0e89cfe3714e11c332c9" Nov 03 09:00:04.752809 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:04.752787 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9_odh-ai-gateway-infra(43dcb009-df45-4a99-94fa-5e427b693335)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" podUID="43dcb009-df45-4a99-94fa-5e427b693335" Nov 03 09:00:15.197051 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:15.197023 5015 scope.go:122] "RemoveContainer" containerID="983704bbc6a9c78aefb8fa9c19140d0d12d8dd0ce9cde5e97e34499e480d6dff" Nov 03 09:00:15.834564 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:15.834531 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" event={"ID":"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0","Type":"ContainerStarted","Data":"626fc1698c4a332f6501f0c1260c4f18109c23c3d914c84dece6f954c0842dbc"} Nov 03 09:00:16.860601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:16.860553 5015 generic.go:356] "Generic (PLEG): container finished" podID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerID="626fc1698c4a332f6501f0c1260c4f18109c23c3d914c84dece6f954c0842dbc" exitCode=60 Nov 03 09:00:16.861078 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:16.860625 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" event={"ID":"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0","Type":"ContainerDied","Data":"626fc1698c4a332f6501f0c1260c4f18109c23c3d914c84dece6f954c0842dbc"} Nov 03 09:00:16.861078 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:16.860743 5015 scope.go:122] "RemoveContainer" containerID="983704bbc6a9c78aefb8fa9c19140d0d12d8dd0ce9cde5e97e34499e480d6dff" Nov 03 09:00:16.861167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:16.861133 5015 scope.go:122] "RemoveContainer" containerID="626fc1698c4a332f6501f0c1260c4f18109c23c3d914c84dece6f954c0842dbc" Nov 03 09:00:16.861514 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:16.861453 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894940-h7rmf_odh-ai-gateway-infra(50d6cdf2-c15b-4be8-ba7a-90807a0a27a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" Nov 03 09:00:16.902118 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:16.902055 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:00:16.969775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:16.969745 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf"] Nov 03 09:00:17.200973 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:17.200900 5015 scope.go:122] "RemoveContainer" containerID="60bc2971ba7b7afb8e03084e3eb8d207fe4bf93804ac0e89cfe3714e11c332c9" Nov 03 09:00:17.872097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:17.872058 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" event={"ID":"43dcb009-df45-4a99-94fa-5e427b693335","Type":"ContainerStarted","Data":"d520b3529a81ca4650789ea44ee570bfd9b96d2b73c7eddf8afb77fb2b17bce9"} Nov 03 09:00:18.019779 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.019756 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" Nov 03 09:00:18.093419 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.093388 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-maas-api-service-ca\") pod \"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0\" (UID: \"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0\") " Nov 03 09:00:18.093622 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.093572 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-kube-api-access-pbrlx\" (UniqueName: \"kubernetes.io/projected/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-kube-api-access-pbrlx\") pod \"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0\" (UID: \"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0\") " Nov 03 09:00:18.093747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.093724 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-maas-api-service-ca" pod "50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" (UID: "50d6cdf2-c15b-4be8-ba7a-90807a0a27a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:00:18.093992 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.093975 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:00:18.095700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.095680 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-kube-api-access-pbrlx" pod "50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" (UID: "50d6cdf2-c15b-4be8-ba7a-90807a0a27a0"). InnerVolumeSpecName "kube-api-access-pbrlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:00:18.195262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.195231 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbrlx\" (UniqueName: \"kubernetes.io/projected/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0-kube-api-access-pbrlx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:00:18.197204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.197187 5015 scope.go:122] "RemoveContainer" containerID="754cba415eb68b5c85e1c8d1d055f4a47d588f9575b117754980b0dcbc9743a5" Nov 03 09:00:18.883250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.883213 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" event={"ID":"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac","Type":"ContainerStarted","Data":"ef44055e875199a9e0d714d4dada8bf9336e865700a5d27cbb9c047a5bda87f1"} Nov 03 09:00:18.884691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.884656 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" Nov 03 09:00:18.884691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.884674 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf" event={"ID":"50d6cdf2-c15b-4be8-ba7a-90807a0a27a0","Type":"ContainerDied","Data":"8b936c395ad42a380b503196b1b6f13970c18df093f6fbb1a261951554f664aa"} Nov 03 09:00:18.884902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.884729 5015 scope.go:122] "RemoveContainer" containerID="626fc1698c4a332f6501f0c1260c4f18109c23c3d914c84dece6f954c0842dbc" Nov 03 09:00:18.886935 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.886910 5015 generic.go:356] "Generic (PLEG): container finished" podID="43dcb009-df45-4a99-94fa-5e427b693335" containerID="d520b3529a81ca4650789ea44ee570bfd9b96d2b73c7eddf8afb77fb2b17bce9" exitCode=60 Nov 03 09:00:18.887079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.886983 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" event={"ID":"43dcb009-df45-4a99-94fa-5e427b693335","Type":"ContainerDied","Data":"d520b3529a81ca4650789ea44ee570bfd9b96d2b73c7eddf8afb77fb2b17bce9"} Nov 03 09:00:18.887380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.887362 5015 scope.go:122] "RemoveContainer" containerID="d520b3529a81ca4650789ea44ee570bfd9b96d2b73c7eddf8afb77fb2b17bce9" Nov 03 09:00:18.887725 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:18.887704 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9_odh-ai-gateway-infra(43dcb009-df45-4a99-94fa-5e427b693335)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" podUID="43dcb009-df45-4a99-94fa-5e427b693335" Nov 03 09:00:18.895469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:18.895452 5015 scope.go:122] "RemoveContainer" containerID="60bc2971ba7b7afb8e03084e3eb8d207fe4bf93804ac0e89cfe3714e11c332c9" Nov 03 09:00:19.117171 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:19.117135 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9"] Nov 03 09:00:19.120518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:19.120488 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf"] Nov 03 09:00:19.126653 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:19.126630 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894940-h7rmf"] Nov 03 09:00:19.204583 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:19.204557 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" path="/var/lib/kubelet/pods/50d6cdf2-c15b-4be8-ba7a-90807a0a27a0/volumes" Nov 03 09:00:19.897019 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:19.896988 5015 generic.go:356] "Generic (PLEG): container finished" podID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerID="ef44055e875199a9e0d714d4dada8bf9336e865700a5d27cbb9c047a5bda87f1" exitCode=60 Nov 03 09:00:19.897387 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:19.897054 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" event={"ID":"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac","Type":"ContainerDied","Data":"ef44055e875199a9e0d714d4dada8bf9336e865700a5d27cbb9c047a5bda87f1"} Nov 03 09:00:19.897387 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:19.897114 5015 scope.go:122] "RemoveContainer" containerID="754cba415eb68b5c85e1c8d1d055f4a47d588f9575b117754980b0dcbc9743a5" Nov 03 09:00:19.897545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:19.897467 5015 scope.go:122] "RemoveContainer" containerID="ef44055e875199a9e0d714d4dada8bf9336e865700a5d27cbb9c047a5bda87f1" Nov 03 09:00:19.897851 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:19.897800 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4_odh-ai-gateway-infra(462c25e5-1a1c-4e7d-bc0a-c650e15f1dac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" Nov 03 09:00:20.016370 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:20.016331 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4"] Nov 03 09:00:20.049227 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:20.049166 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" Nov 03 09:00:20.116774 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:20.116746 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43dcb009-df45-4a99-94fa-5e427b693335-kube-api-access-cbd99\" (UniqueName: \"kubernetes.io/projected/43dcb009-df45-4a99-94fa-5e427b693335-kube-api-access-cbd99\") pod \"43dcb009-df45-4a99-94fa-5e427b693335\" (UID: \"43dcb009-df45-4a99-94fa-5e427b693335\") " Nov 03 09:00:20.116971 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:20.116796 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43dcb009-df45-4a99-94fa-5e427b693335-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43dcb009-df45-4a99-94fa-5e427b693335-maas-api-service-ca\") pod \"43dcb009-df45-4a99-94fa-5e427b693335\" (UID: \"43dcb009-df45-4a99-94fa-5e427b693335\") " Nov 03 09:00:20.117167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:20.117142 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43dcb009-df45-4a99-94fa-5e427b693335-maas-api-service-ca" pod "43dcb009-df45-4a99-94fa-5e427b693335" (UID: "43dcb009-df45-4a99-94fa-5e427b693335"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:00:20.119136 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:20.119072 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43dcb009-df45-4a99-94fa-5e427b693335-kube-api-access-cbd99" pod "43dcb009-df45-4a99-94fa-5e427b693335" (UID: "43dcb009-df45-4a99-94fa-5e427b693335"). InnerVolumeSpecName "kube-api-access-cbd99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:00:20.217809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:20.217777 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbd99\" (UniqueName: \"kubernetes.io/projected/43dcb009-df45-4a99-94fa-5e427b693335-kube-api-access-cbd99\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:00:20.217809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:20.217807 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43dcb009-df45-4a99-94fa-5e427b693335-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:00:20.908076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:20.908052 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" Nov 03 09:00:20.908460 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:20.908049 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9" event={"ID":"43dcb009-df45-4a99-94fa-5e427b693335","Type":"ContainerDied","Data":"e8b69dec8b46e7dc18a7eeaf9f7555d5fb027f02ce4ac1ee43ddcdd7cee39cd5"} Nov 03 09:00:20.908460 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:20.908201 5015 scope.go:122] "RemoveContainer" containerID="d520b3529a81ca4650789ea44ee570bfd9b96d2b73c7eddf8afb77fb2b17bce9" Nov 03 09:00:20.971265 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:20.971238 5015 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43dcb009_df45_4a99_94fa_5e427b693335.slice\": RecentStats: unable to find data in memory cache]" Nov 03 09:00:21.072909 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.072881 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" Nov 03 09:00:21.126690 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.126655 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-kube-api-access-qhntn\" (UniqueName: \"kubernetes.io/projected/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-kube-api-access-qhntn\") pod \"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac\" (UID: \"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac\") " Nov 03 09:00:21.126859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.126824 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-maas-api-service-ca\") pod \"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac\" (UID: \"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac\") " Nov 03 09:00:21.127160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.127133 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-maas-api-service-ca" pod "462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" (UID: "462c25e5-1a1c-4e7d-bc0a-c650e15f1dac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:00:21.127330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.127313 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:00:21.128889 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.128867 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-kube-api-access-qhntn" pod "462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" (UID: "462c25e5-1a1c-4e7d-bc0a-c650e15f1dac"). InnerVolumeSpecName "kube-api-access-qhntn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:00:21.146644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.146623 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9"] Nov 03 09:00:21.170797 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.170733 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894940-nq2l9"] Nov 03 09:00:21.209330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.209295 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43dcb009-df45-4a99-94fa-5e427b693335" path="/var/lib/kubelet/pods/43dcb009-df45-4a99-94fa-5e427b693335/volumes" Nov 03 09:00:21.228286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.228255 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhntn\" (UniqueName: \"kubernetes.io/projected/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac-kube-api-access-qhntn\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:00:21.922407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.922365 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" event={"ID":"462c25e5-1a1c-4e7d-bc0a-c650e15f1dac","Type":"ContainerDied","Data":"e09d015eeb8ada132402653e73201b8634d1fec8727e1dcc337593f70d77f75d"} Nov 03 09:00:21.922407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.922391 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4" Nov 03 09:00:21.922981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:21.922428 5015 scope.go:122] "RemoveContainer" containerID="ef44055e875199a9e0d714d4dada8bf9336e865700a5d27cbb9c047a5bda87f1" Nov 03 09:00:22.093227 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:22.093191 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4"] Nov 03 09:00:22.098161 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:22.098130 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894940-qqkz4"] Nov 03 09:00:23.204774 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:23.204744 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" path="/var/lib/kubelet/pods/462c25e5-1a1c-4e7d-bc0a-c650e15f1dac/volumes" Nov 03 09:00:32.207457 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.207424 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k"] Nov 03 09:00:32.208261 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208234 5015 cpu_manager.go:414] "containerMap: removing container" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerName="cleanup" Nov 03 09:00:32.208389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208262 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerName="cleanup" Nov 03 09:00:32.208389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208290 5015 cpu_manager.go:414] "containerMap: removing container" podUID="43dcb009-df45-4a99-94fa-5e427b693335" containerName="cleanup" Nov 03 09:00:32.208389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208298 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43dcb009-df45-4a99-94fa-5e427b693335" containerName="cleanup" Nov 03 09:00:32.208389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208321 5015 cpu_manager.go:414] "containerMap: removing container" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerName="cleanup" Nov 03 09:00:32.208389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208328 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerName="cleanup" Nov 03 09:00:32.208389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208341 5015 cpu_manager.go:414] "containerMap: removing container" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerName="cleanup" Nov 03 09:00:32.208389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208348 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerName="cleanup" Nov 03 09:00:32.208389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208366 5015 cpu_manager.go:414] "containerMap: removing container" podUID="43dcb009-df45-4a99-94fa-5e427b693335" containerName="cleanup" Nov 03 09:00:32.208389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208373 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43dcb009-df45-4a99-94fa-5e427b693335" containerName="cleanup" Nov 03 09:00:32.208389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208388 5015 cpu_manager.go:414] "containerMap: removing container" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerName="cleanup" Nov 03 09:00:32.208769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208396 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerName="cleanup" Nov 03 09:00:32.208769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208411 5015 cpu_manager.go:414] "containerMap: removing container" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerName="cleanup" Nov 03 09:00:32.208769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208418 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerName="cleanup" Nov 03 09:00:32.208769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208598 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerName="cleanup" Nov 03 09:00:32.208769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208611 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerName="cleanup" Nov 03 09:00:32.208769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208627 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="43dcb009-df45-4a99-94fa-5e427b693335" containerName="cleanup" Nov 03 09:00:32.208769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208637 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerName="cleanup" Nov 03 09:00:32.208769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208644 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="43dcb009-df45-4a99-94fa-5e427b693335" containerName="cleanup" Nov 03 09:00:32.208769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208656 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerName="cleanup" Nov 03 09:00:32.208769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.208668 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="43dcb009-df45-4a99-94fa-5e427b693335" containerName="cleanup" Nov 03 09:00:32.213948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.213927 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.306789 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.306753 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k"] Nov 03 09:00:32.358536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.358502 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.358728 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.358545 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.358855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.358831 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.359003 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.358979 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.359077 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.359061 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.359119 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.359091 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.359303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.359284 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.359367 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.359322 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74mt6\" (UniqueName: \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-kube-api-access-74mt6\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.359513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.359465 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.461109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461023 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.461109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461062 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.461109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461095 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.461404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461217 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.461404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461280 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.461599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461575 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.461738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461714 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.461738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461652 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.461738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461596 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.461738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461761 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.462099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461803 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mt6\" (UniqueName: \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-kube-api-access-74mt6\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.462099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461879 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.462099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.461904 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.462099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.462066 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.464113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.464085 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.464253 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.464230 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.522606 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.522578 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.522790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.522773 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74mt6\" (UniqueName: \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-kube-api-access-74mt6\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.528296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.528271 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:32.972608 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:00:32.972561 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1513d8_3e7c_4733_aa56_9ccd7eeac09c.slice/crio-2f8079178d1e91b64cb2a053480f10c8aa48e26fa8808418246770211683ae06 WatchSource:0}: Error finding container 2f8079178d1e91b64cb2a053480f10c8aa48e26fa8808418246770211683ae06: Status 404 returned error can't find the container with id 2f8079178d1e91b64cb2a053480f10c8aa48e26fa8808418246770211683ae06 Nov 03 09:00:32.973357 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.973331 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k"] Nov 03 09:00:32.975111 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.975075 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:00:32.975217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.975159 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:00:32.975217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:32.975204 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:00:33.008113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:33.008087 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" event={"ID":"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c","Type":"ContainerStarted","Data":"2f8079178d1e91b64cb2a053480f10c8aa48e26fa8808418246770211683ae06"} Nov 03 09:00:34.018560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:34.018526 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" event={"ID":"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c","Type":"ContainerStarted","Data":"85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389"} Nov 03 09:00:34.190788 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:34.190734 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" podStartSLOduration=2.19072024 podStartE2EDuration="2.19072024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:00:34.189111224 +0000 UTC m=+1267.114053446" watchObservedRunningTime="2026-11-03 09:00:34.19072024 +0000 UTC m=+1267.115662456" Nov 03 09:00:34.530656 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:34.530608 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:34.534536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:34.533529 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 09:00:34.534536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:34.533598 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 09:00:35.528989 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:35.528953 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 09:00:35.529469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:35.529427 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 09:00:36.529143 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:36.529105 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 09:00:36.529598 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:36.529176 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 09:00:37.528634 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:37.528601 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 09:00:37.528825 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:37.528666 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 09:00:38.529536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:38.529463 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 09:00:38.529949 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:38.529549 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 09:00:39.528680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:39.528648 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 09:00:39.528867 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:39.528699 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 09:00:40.533131 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:40.533098 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:40.533584 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:40.533515 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:40.534292 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:40.534275 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:00:46.901810 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:00:46.901772 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:00:59.025713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.025681 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd"] Nov 03 09:00:59.026212 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.026198 5015 cpu_manager.go:414] "containerMap: removing container" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerName="cleanup" Nov 03 09:00:59.026212 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.026209 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerName="cleanup" Nov 03 09:00:59.026296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.026240 5015 cpu_manager.go:414] "containerMap: removing container" podUID="43dcb009-df45-4a99-94fa-5e427b693335" containerName="cleanup" Nov 03 09:00:59.026296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.026245 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43dcb009-df45-4a99-94fa-5e427b693335" containerName="cleanup" Nov 03 09:00:59.026365 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.026356 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d6cdf2-c15b-4be8-ba7a-90807a0a27a0" containerName="cleanup" Nov 03 09:00:59.026414 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.026366 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="462c25e5-1a1c-4e7d-bc0a-c650e15f1dac" containerName="cleanup" Nov 03 09:00:59.031102 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.031069 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:00:59.086916 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.086881 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd"] Nov 03 09:00:59.125053 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.125017 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8bfa982-lnrcd\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:00:59.125231 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.125205 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz476\" (UniqueName: \"kubernetes.io/projected/c79283dd-fcdb-4c11-9796-9deb0f836c91-kube-api-access-pz476\") pod \"maas-api-revoke-sub-f8bfa982-lnrcd\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:00:59.125352 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.125308 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8bfa982-lnrcd\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:00:59.133052 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.133029 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 09:00:59.133135 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.133094 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:00:59.226374 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.226341 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8bfa982-lnrcd\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:00:59.226583 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.226433 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8bfa982-lnrcd\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:00:59.226583 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.226545 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz476\" (UniqueName: \"kubernetes.io/projected/c79283dd-fcdb-4c11-9796-9deb0f836c91-kube-api-access-pz476\") pod \"maas-api-revoke-sub-f8bfa982-lnrcd\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:00:59.227268 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.227241 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8bfa982-lnrcd\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:00:59.229041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.229016 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8bfa982-lnrcd\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:00:59.286445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.286375 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz476\" (UniqueName: \"kubernetes.io/projected/c79283dd-fcdb-4c11-9796-9deb0f836c91-kube-api-access-pz476\") pod \"maas-api-revoke-sub-f8bfa982-lnrcd\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:00:59.353485 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.353449 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:00:59.732603 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:00:59.732403 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd"] Nov 03 09:00:59.736529 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:00:59.736497 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79283dd_fcdb_4c11_9796_9deb0f836c91.slice/crio-f9996d9e5ac911ee52ef5e86c92f04c7275795504a622a0907c4b5c806cfb04a WatchSource:0}: Error finding container f9996d9e5ac911ee52ef5e86c92f04c7275795504a622a0907c4b5c806cfb04a: Status 404 returned error can't find the container with id f9996d9e5ac911ee52ef5e86c92f04c7275795504a622a0907c4b5c806cfb04a Nov 03 09:01:00.242227 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:00.242189 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" event={"ID":"c79283dd-fcdb-4c11-9796-9deb0f836c91","Type":"ContainerStarted","Data":"bc95a089bf6211f59b7e0ac61baaa2efd07c21514c6a9806b1ea299f830546d4"} Nov 03 09:01:00.242716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:00.242250 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" event={"ID":"c79283dd-fcdb-4c11-9796-9deb0f836c91","Type":"ContainerStarted","Data":"f9996d9e5ac911ee52ef5e86c92f04c7275795504a622a0907c4b5c806cfb04a"} Nov 03 09:01:00.360739 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:00.360618 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" podStartSLOduration=2.360604674 podStartE2EDuration="2.360604674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:00.357782224 +0000 UTC m=+1293.282724441" watchObservedRunningTime="2026-11-03 09:01:00.360604674 +0000 UTC m=+1293.285547119" Nov 03 09:01:01.254437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:01.254398 5015 generic.go:356] "Generic (PLEG): container finished" podID="c79283dd-fcdb-4c11-9796-9deb0f836c91" containerID="bc95a089bf6211f59b7e0ac61baaa2efd07c21514c6a9806b1ea299f830546d4" exitCode=0 Nov 03 09:01:01.255247 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:01.255176 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" event={"ID":"c79283dd-fcdb-4c11-9796-9deb0f836c91","Type":"ContainerDied","Data":"bc95a089bf6211f59b7e0ac61baaa2efd07c21514c6a9806b1ea299f830546d4"} Nov 03 09:01:02.411642 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:02.411612 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:01:02.566588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:02.566504 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-openshift-ca\") pod \"c79283dd-fcdb-4c11-9796-9deb0f836c91\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " Nov 03 09:01:02.566752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:02.566622 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79283dd-fcdb-4c11-9796-9deb0f836c91-kube-api-access-pz476\" (UniqueName: \"kubernetes.io/projected/c79283dd-fcdb-4c11-9796-9deb0f836c91-kube-api-access-pz476\") pod \"c79283dd-fcdb-4c11-9796-9deb0f836c91\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " Nov 03 09:01:02.566752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:02.566683 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-xks-ca\") pod \"c79283dd-fcdb-4c11-9796-9deb0f836c91\" (UID: \"c79283dd-fcdb-4c11-9796-9deb0f836c91\") " Nov 03 09:01:02.567152 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:02.566855 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-openshift-ca" pod "c79283dd-fcdb-4c11-9796-9deb0f836c91" (UID: "c79283dd-fcdb-4c11-9796-9deb0f836c91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:01:02.567284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:02.567162 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:02.569447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:02.569408 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79283dd-fcdb-4c11-9796-9deb0f836c91-kube-api-access-pz476" pod "c79283dd-fcdb-4c11-9796-9deb0f836c91" (UID: "c79283dd-fcdb-4c11-9796-9deb0f836c91"). InnerVolumeSpecName "kube-api-access-pz476". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:01:02.581676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:02.581645 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-xks-ca" pod "c79283dd-fcdb-4c11-9796-9deb0f836c91" (UID: "c79283dd-fcdb-4c11-9796-9deb0f836c91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:01:02.668772 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:02.668736 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz476\" (UniqueName: \"kubernetes.io/projected/c79283dd-fcdb-4c11-9796-9deb0f836c91-kube-api-access-pz476\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:02.668772 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:02.668771 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79283dd-fcdb-4c11-9796-9deb0f836c91-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:03.271773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:03.271741 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" Nov 03 09:01:03.271935 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:03.271748 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd" event={"ID":"c79283dd-fcdb-4c11-9796-9deb0f836c91","Type":"ContainerDied","Data":"f9996d9e5ac911ee52ef5e86c92f04c7275795504a622a0907c4b5c806cfb04a"} Nov 03 09:01:03.271935 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:03.271869 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9996d9e5ac911ee52ef5e86c92f04c7275795504a622a0907c4b5c806cfb04a" Nov 03 09:01:11.851016 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.850936 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g"] Nov 03 09:01:11.851559 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.851537 5015 cpu_manager.go:414] "containerMap: removing container" podUID="c79283dd-fcdb-4c11-9796-9deb0f836c91" containerName="revoke-keys" Nov 03 09:01:11.851559 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.851555 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c79283dd-fcdb-4c11-9796-9deb0f836c91" containerName="revoke-keys" Nov 03 09:01:11.851700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.851690 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="c79283dd-fcdb-4c11-9796-9deb0f836c91" containerName="revoke-keys" Nov 03 09:01:11.855387 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.855369 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:11.956524 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.954537 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g"] Nov 03 09:01:11.967852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.967819 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:11.968031 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.967865 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc7a494-d07b-4693-bc27-31359cb34014-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:11.968092 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.968058 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc7a494-d07b-4693-bc27-31359cb34014-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:11.968153 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.968103 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:11.968205 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.968169 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:11.968258 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.968206 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:11.968310 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.968283 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv7n2\" (UniqueName: \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-kube-api-access-zv7n2\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:11.968416 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.968307 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:11.968416 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:11.968376 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.069418 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.069388 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc7a494-d07b-4693-bc27-31359cb34014-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.069639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.069429 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.069639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.069462 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.069639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.069524 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.069639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.069579 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv7n2\" (UniqueName: \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-kube-api-access-zv7n2\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.069639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.069615 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.069925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.069683 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.069925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.069726 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.069925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.069764 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc7a494-d07b-4693-bc27-31359cb34014-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.070400 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.070317 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc7a494-d07b-4693-bc27-31359cb34014-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.070400 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.070371 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.070920 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.070695 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.070920 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.070902 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.071091 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.070765 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.072277 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.072255 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.072623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.072605 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc7a494-d07b-4693-bc27-31359cb34014-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.129537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.129454 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.129912 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.129895 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv7n2\" (UniqueName: \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-kube-api-access-zv7n2\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.169532 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.169504 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:12.608082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.608049 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g"] Nov 03 09:01:12.645423 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:01:12.645393 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc7a494_d07b_4693_bc27_31359cb34014.slice/crio-5c77d0aad7bafd2860174fe6461bfdcaebb3e5922edd58c0cc1a45c52d5785d2 WatchSource:0}: Error finding container 5c77d0aad7bafd2860174fe6461bfdcaebb3e5922edd58c0cc1a45c52d5785d2: Status 404 returned error can't find the container with id 5c77d0aad7bafd2860174fe6461bfdcaebb3e5922edd58c0cc1a45c52d5785d2 Nov 03 09:01:12.647712 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.647679 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:01:12.647829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.647748 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:01:12.647829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:12.647779 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:01:13.371891 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:13.371818 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" event={"ID":"1bc7a494-d07b-4693-bc27-31359cb34014","Type":"ContainerStarted","Data":"5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a"} Nov 03 09:01:13.372431 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:13.371909 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" event={"ID":"1bc7a494-d07b-4693-bc27-31359cb34014","Type":"ContainerStarted","Data":"5c77d0aad7bafd2860174fe6461bfdcaebb3e5922edd58c0cc1a45c52d5785d2"} Nov 03 09:01:13.495638 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:13.495590 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" podStartSLOduration=2.495577099 podStartE2EDuration="2.495577099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:13.492131605 +0000 UTC m=+1306.417073832" watchObservedRunningTime="2026-11-03 09:01:13.495577099 +0000 UTC m=+1306.420519315" Nov 03 09:01:14.170714 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:14.170685 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:14.172045 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:14.172024 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 09:01:14.172145 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:14.172076 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 09:01:15.172029 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:15.170599 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 09:01:15.172561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:15.172057 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 09:01:16.170433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:16.170395 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 09:01:16.170649 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:16.170446 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 09:01:16.901789 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:01:16.901748 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:01:17.170140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:17.170060 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 09:01:17.170140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:17.170113 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 09:01:18.170737 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:18.170704 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 09:01:18.171212 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:18.171187 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 09:01:19.170192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:19.170146 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 09:01:19.170401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:19.170209 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 09:01:20.174949 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:20.174870 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:20.175404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:20.175330 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:20.176014 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:20.175993 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:01:29.055099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:29.055067 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8"] Nov 03 09:01:29.105538 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:29.105499 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-746ab303-8qpf8"] Nov 03 09:01:29.206222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:29.206187 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5552d74-e374-456b-9d6b-8b4d1afb58f5" path="/var/lib/kubelet/pods/a5552d74-e374-456b-9d6b-8b4d1afb58f5/volumes" Nov 03 09:01:31.050069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.050029 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq"] Nov 03 09:01:31.056417 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.056396 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.110512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.110450 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq"] Nov 03 09:01:31.221351 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.221321 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-provider-credentials\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.221541 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.221451 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dk7k\" (UniqueName: \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-kube-api-access-6dk7k\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.221541 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.221535 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-routing-overlay-volume\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.221622 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.221596 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-plugins-config-volume\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.322433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.322346 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dk7k\" (UniqueName: \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-kube-api-access-6dk7k\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.322591 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.322464 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-routing-overlay-volume\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.322591 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.322542 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-plugins-config-volume\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.322709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.322695 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-provider-credentials\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.323244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.323219 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-routing-overlay-volume\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.323395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.323350 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-plugins-config-volume\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.325203 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.325186 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-provider-credentials\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.382666 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.382633 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dk7k\" (UniqueName: \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-kube-api-access-6dk7k\") pod \"payload-processing-external-model-6845cdf9f-6sxqq\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:31.670947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:31.670859 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:32.078637 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:01:32.078605 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b310f2a_9ee6_4550_a5f6_6edd579070b1.slice/crio-7bf6f0763a2ffc8bd9927c91c9bae3d4ba7f4ea2ca32aac9e9ccc4e191fa5fe2 WatchSource:0}: Error finding container 7bf6f0763a2ffc8bd9927c91c9bae3d4ba7f4ea2ca32aac9e9ccc4e191fa5fe2: Status 404 returned error can't find the container with id 7bf6f0763a2ffc8bd9927c91c9bae3d4ba7f4ea2ca32aac9e9ccc4e191fa5fe2 Nov 03 09:01:32.078988 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.078664 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq"] Nov 03 09:01:32.552939 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.552892 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" event={"ID":"7b310f2a-9ee6-4550-a5f6-6edd579070b1","Type":"ContainerStarted","Data":"53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a"} Nov 03 09:01:32.553108 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.552967 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:32.553108 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.552995 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" event={"ID":"7b310f2a-9ee6-4550-a5f6-6edd579070b1","Type":"ContainerStarted","Data":"7bf6f0763a2ffc8bd9927c91c9bae3d4ba7f4ea2ca32aac9e9ccc4e191fa5fe2"} Nov 03 09:01:32.662638 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.662603 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj"] Nov 03 09:01:32.675459 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.675430 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.677403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.677345 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" podStartSLOduration=1.677329334 podStartE2EDuration="1.677329334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:32.67393313 +0000 UTC m=+1325.598875347" watchObservedRunningTime="2026-11-03 09:01:32.677329334 +0000 UTC m=+1325.602271557" Nov 03 09:01:32.743590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.743556 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.743759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.743686 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.743805 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.743756 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.743805 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.743780 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a4006d-88b5-4965-8738-1bd170cc4ede-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.743869 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.743839 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkc2n\" (UniqueName: \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-kube-api-access-lkc2n\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.743869 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.743856 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.743945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.743933 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.743980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.743954 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.744012 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.744006 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.775739 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.775703 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj"] Nov 03 09:01:32.845239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845152 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.845239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845202 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.845505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845264 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.845505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845334 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.845505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845436 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.845746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845520 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.845746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845551 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a4006d-88b5-4965-8738-1bd170cc4ede-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.845746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845611 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.845746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845621 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.845746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845616 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkc2n\" (UniqueName: \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-kube-api-access-lkc2n\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.845746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845671 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.846072 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.845952 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.846072 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.846038 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.846968 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.846938 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a4006d-88b5-4965-8738-1bd170cc4ede-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.848526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.848495 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.848671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.848651 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.904857 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.904827 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.905006 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.904935 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkc2n\" (UniqueName: \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-kube-api-access-lkc2n\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:32.989812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:32.989786 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:33.423436 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:33.423397 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj"] Nov 03 09:01:33.462073 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:01:33.462043 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a4006d_88b5_4965_8738_1bd170cc4ede.slice/crio-3b9e65182973790b112fd8493175d159e27ab4c589a5fe6e21a378faf4529af6 WatchSource:0}: Error finding container 3b9e65182973790b112fd8493175d159e27ab4c589a5fe6e21a378faf4529af6: Status 404 returned error can't find the container with id 3b9e65182973790b112fd8493175d159e27ab4c589a5fe6e21a378faf4529af6 Nov 03 09:01:33.465071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:33.465031 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:01:33.465161 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:33.465113 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:01:33.465161 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:33.465152 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:01:33.564667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:33.564617 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" event={"ID":"b7a4006d-88b5-4965-8738-1bd170cc4ede","Type":"ContainerStarted","Data":"dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302"} Nov 03 09:01:33.564812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:33.564682 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" event={"ID":"b7a4006d-88b5-4965-8738-1bd170cc4ede","Type":"ContainerStarted","Data":"3b9e65182973790b112fd8493175d159e27ab4c589a5fe6e21a378faf4529af6"} Nov 03 09:01:33.686297 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:33.686187 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" podStartSLOduration=1.6861729319999998 podStartE2EDuration="1.686172932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:33.684878283 +0000 UTC m=+1326.609820500" watchObservedRunningTime="2026-11-03 09:01:33.686172932 +0000 UTC m=+1326.611115189" Nov 03 09:01:33.990670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:33.990601 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:34.992530 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:34.992462 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 09:01:34.993050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:34.992564 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" podUID="b7a4006d-88b5-4965-8738-1bd170cc4ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 09:01:35.991112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:35.991081 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 09:01:35.991278 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:35.991133 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" podUID="b7a4006d-88b5-4965-8738-1bd170cc4ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 09:01:36.991063 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:36.991020 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 09:01:36.991458 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:36.991102 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" podUID="b7a4006d-88b5-4965-8738-1bd170cc4ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 09:01:37.991292 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:37.991257 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 09:01:37.991814 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:37.991325 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" podUID="b7a4006d-88b5-4965-8738-1bd170cc4ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 09:01:38.990844 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:38.990812 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 09:01:38.991026 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:38.990864 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" podUID="b7a4006d-88b5-4965-8738-1bd170cc4ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 09:01:39.990192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:39.990157 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 09:01:39.990718 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:39.990219 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" podUID="b7a4006d-88b5-4965-8738-1bd170cc4ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 09:01:40.994221 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:40.994191 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:40.994708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:40.994690 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:40.995468 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:40.995446 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:01:43.416331 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.416292 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2"] Nov 03 09:01:43.420374 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.420357 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:43.487034 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.487002 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6476a5c6-sdhh2\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:43.487235 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.487052 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w82f7\" (UniqueName: \"kubernetes.io/projected/981df4b1-4b75-49e8-9df1-b74be2be6131-kube-api-access-w82f7\") pod \"maas-api-revoke-sub-6476a5c6-sdhh2\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:43.487308 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.487256 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6476a5c6-sdhh2\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:43.525142 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.525105 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 09:01:43.525317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.525260 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:01:43.529240 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.529213 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2"] Nov 03 09:01:43.569252 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.569212 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" podUID="7b310f2a-9ee6-4550-a5f6-6edd579070b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:01:43.588943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.588906 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w82f7\" (UniqueName: \"kubernetes.io/projected/981df4b1-4b75-49e8-9df1-b74be2be6131-kube-api-access-w82f7\") pod \"maas-api-revoke-sub-6476a5c6-sdhh2\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:43.589105 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.589024 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6476a5c6-sdhh2\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:43.589253 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.589235 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6476a5c6-sdhh2\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:43.589944 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.589866 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6476a5c6-sdhh2\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:43.591698 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.591673 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6476a5c6-sdhh2\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:43.649081 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.649051 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w82f7\" (UniqueName: \"kubernetes.io/projected/981df4b1-4b75-49e8-9df1-b74be2be6131-kube-api-access-w82f7\") pod \"maas-api-revoke-sub-6476a5c6-sdhh2\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:43.736503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:43.734375 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:44.012745 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:44.012590 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf"] Nov 03 09:01:44.018362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:44.018333 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq"] Nov 03 09:01:44.020652 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:44.020619 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" podUID="7b310f2a-9ee6-4550-a5f6-6edd579070b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:01:44.154687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:44.154658 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2"] Nov 03 09:01:44.156624 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:01:44.156598 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981df4b1_4b75_49e8_9df1_b74be2be6131.slice/crio-b68e1ca9ca1cd14db49112af6f3625f72e84177cf910980fdff0f7eb99fc985f WatchSource:0}: Error finding container b68e1ca9ca1cd14db49112af6f3625f72e84177cf910980fdff0f7eb99fc985f: Status 404 returned error can't find the container with id b68e1ca9ca1cd14db49112af6f3625f72e84177cf910980fdff0f7eb99fc985f Nov 03 09:01:44.695614 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:44.692819 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" event={"ID":"981df4b1-4b75-49e8-9df1-b74be2be6131","Type":"ContainerStarted","Data":"c0af727515840d8d6931586fc62ec416bc2fc7b156b213d8994c8d381cb96703"} Nov 03 09:01:44.695614 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:44.692957 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" event={"ID":"981df4b1-4b75-49e8-9df1-b74be2be6131","Type":"ContainerStarted","Data":"b68e1ca9ca1cd14db49112af6f3625f72e84177cf910980fdff0f7eb99fc985f"} Nov 03 09:01:44.822079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:44.821139 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" podStartSLOduration=1.821119243 podStartE2EDuration="1.821119243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:44.81183409 +0000 UTC m=+1337.736776320" watchObservedRunningTime="2026-11-03 09:01:44.821119243 +0000 UTC m=+1337.746061462" Nov 03 09:01:45.739526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:45.737201 5015 generic.go:356] "Generic (PLEG): container finished" podID="981df4b1-4b75-49e8-9df1-b74be2be6131" containerID="c0af727515840d8d6931586fc62ec416bc2fc7b156b213d8994c8d381cb96703" exitCode=0 Nov 03 09:01:45.739526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:45.737413 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" event={"ID":"981df4b1-4b75-49e8-9df1-b74be2be6131","Type":"ContainerDied","Data":"c0af727515840d8d6931586fc62ec416bc2fc7b156b213d8994c8d381cb96703"} Nov 03 09:01:46.656503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:46.649623 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:46.931561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:46.929562 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:47.098700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:47.098632 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podUID="21ad0cfe-e3c6-4770-b0b3-7c512e7f1830" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:47.164905 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:47.164863 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:47.307554 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:01:47.300743 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:01:47.449919 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:47.447992 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 09:01:47.503504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:47.497541 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:47.661637 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:47.655200 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 09:01:47.728623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:47.727944 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:47.932561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:47.930576 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 03 09:01:48.047537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.043561 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 09:01:48.047537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.043626 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 09:01:48.163513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.159645 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podUID="21ad0cfe-e3c6-4770-b0b3-7c512e7f1830" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 09:01:48.163513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.160958 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 09:01:48.194073 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.193300 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 09:01:48.246056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.216258 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 09:01:48.399518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.385857 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:48.454583 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.448603 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 09:01:48.454583 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.448961 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 09:01:48.549931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.549423 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:01:48.549931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.549544 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:48.619696 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.618428 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981df4b1-4b75-49e8-9df1-b74be2be6131-kube-api-access-w82f7\" (UniqueName: \"kubernetes.io/projected/981df4b1-4b75-49e8-9df1-b74be2be6131-kube-api-access-w82f7\") pod \"981df4b1-4b75-49e8-9df1-b74be2be6131\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " Nov 03 09:01:48.619696 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.618549 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-openshift-ca\") pod \"981df4b1-4b75-49e8-9df1-b74be2be6131\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " Nov 03 09:01:48.619696 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.618728 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-xks-ca\") pod \"981df4b1-4b75-49e8-9df1-b74be2be6131\" (UID: \"981df4b1-4b75-49e8-9df1-b74be2be6131\") " Nov 03 09:01:48.655636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.655590 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:48.656854 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.656817 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:48.657816 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.657798 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 09:01:48.791465 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.791418 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Nov 03 09:01:48.845730 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.845690 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-openshift-ca" pod "981df4b1-4b75-49e8-9df1-b74be2be6131" (UID: "981df4b1-4b75-49e8-9df1-b74be2be6131"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:01:48.879639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.847721 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:01:48.979568 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.979529 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:49.013507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:48.995322 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:49.021542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.014340 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" podUID="c4bf6832-6716-4958-82f5-fe4c3340090b" containerName="payload-processing" containerID="cri-o://4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c" gracePeriod=25 Nov 03 09:01:49.074034 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.022128 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" podUID="7b310f2a-9ee6-4550-a5f6-6edd579070b1" containerName="payload-processing" containerID="cri-o://53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a" gracePeriod=25 Nov 03 09:01:49.074034 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.023186 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-xks-ca" pod "981df4b1-4b75-49e8-9df1-b74be2be6131" (UID: "981df4b1-4b75-49e8-9df1-b74be2be6131"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:01:49.103335 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.103295 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981df4b1-4b75-49e8-9df1-b74be2be6131-kube-api-access-w82f7" pod "981df4b1-4b75-49e8-9df1-b74be2be6131" (UID: "981df4b1-4b75-49e8-9df1-b74be2be6131"). InnerVolumeSpecName "kube-api-access-w82f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:01:49.196506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.188500 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981df4b1-4b75-49e8-9df1-b74be2be6131-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:49.196506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.188542 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w82f7\" (UniqueName: \"kubernetes.io/projected/981df4b1-4b75-49e8-9df1-b74be2be6131-kube-api-access-w82f7\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:49.196506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.190200 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podUID="21ad0cfe-e3c6-4770-b0b3-7c512e7f1830" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 09:01:49.196506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.190582 5015 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Nov 03 09:01:49.431335 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.363407 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 09:01:49.508510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.473626 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:49.728560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.728019 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:49.796033 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.795997 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Nov 03 09:01:49.897187 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.897159 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" Nov 03 09:01:49.920334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.920297 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:49.997269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.995018 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 09:01:49.997269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.995063 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2" event={"ID":"981df4b1-4b75-49e8-9df1-b74be2be6131","Type":"ContainerDied","Data":"b68e1ca9ca1cd14db49112af6f3625f72e84177cf910980fdff0f7eb99fc985f"} Nov 03 09:01:49.997269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.995100 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b68e1ca9ca1cd14db49112af6f3625f72e84177cf910980fdff0f7eb99fc985f" Nov 03 09:01:49.997269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.995154 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 09:01:49.998734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:49.997963 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:50.006860 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.006836 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 09:01:50.028052 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.011681 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 09:01:50.028052 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.016185 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:01:50.031218 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.029007 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 09:01:50.037953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.037082 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 09:01:50.227649 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.227625 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:50.231053 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.231032 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 09:01:50.305932 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.305836 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-kube-api-access-qf9wz\" (UniqueName: \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-kube-api-access-qf9wz\") pod \"c4bf6832-6716-4958-82f5-fe4c3340090b\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " Nov 03 09:01:50.305932 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.305877 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-routing-overlay-volume\") pod \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " Nov 03 09:01:50.305932 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.305925 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-plugins-config-volume\") pod \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " Nov 03 09:01:50.306218 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.306095 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-kube-api-access-6dk7k\" (UniqueName: \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-kube-api-access-6dk7k\") pod \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " Nov 03 09:01:50.306218 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.306161 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-routing-overlay-volume\") pod \"c4bf6832-6716-4958-82f5-fe4c3340090b\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " Nov 03 09:01:50.306342 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.306274 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-plugins-config-volume" pod "7b310f2a-9ee6-4550-a5f6-6edd579070b1" (UID: "7b310f2a-9ee6-4550-a5f6-6edd579070b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:01:50.306342 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.306303 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-provider-credentials\") pod \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\" (UID: \"7b310f2a-9ee6-4550-a5f6-6edd579070b1\") " Nov 03 09:01:50.306342 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.306323 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-routing-overlay-volume" pod "7b310f2a-9ee6-4550-a5f6-6edd579070b1" (UID: "7b310f2a-9ee6-4550-a5f6-6edd579070b1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:01:50.306519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.306426 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-provider-credentials\") pod \"c4bf6832-6716-4958-82f5-fe4c3340090b\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " Nov 03 09:01:50.306519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.306461 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-plugins-config-volume\") pod \"c4bf6832-6716-4958-82f5-fe4c3340090b\" (UID: \"c4bf6832-6716-4958-82f5-fe4c3340090b\") " Nov 03 09:01:50.306689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.306667 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-routing-overlay-volume" pod "c4bf6832-6716-4958-82f5-fe4c3340090b" (UID: "c4bf6832-6716-4958-82f5-fe4c3340090b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:01:50.307235 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.307209 5015 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-routing-overlay-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:50.307349 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.307240 5015 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-routing-overlay-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:50.307349 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.307258 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b310f2a-9ee6-4550-a5f6-6edd579070b1-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:50.307465 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.307340 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-plugins-config-volume" pod "c4bf6832-6716-4958-82f5-fe4c3340090b" (UID: "c4bf6832-6716-4958-82f5-fe4c3340090b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:01:50.308575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.308548 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-kube-api-access-qf9wz" pod "c4bf6832-6716-4958-82f5-fe4c3340090b" (UID: "c4bf6832-6716-4958-82f5-fe4c3340090b"). InnerVolumeSpecName "kube-api-access-qf9wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:01:50.308834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.308808 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-kube-api-access-6dk7k" pod "7b310f2a-9ee6-4550-a5f6-6edd579070b1" (UID: "7b310f2a-9ee6-4550-a5f6-6edd579070b1"). InnerVolumeSpecName "kube-api-access-6dk7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:01:50.309233 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.309215 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-provider-credentials" pod "7b310f2a-9ee6-4550-a5f6-6edd579070b1" (UID: "7b310f2a-9ee6-4550-a5f6-6edd579070b1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:01:50.309618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.309599 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-provider-credentials" pod "c4bf6832-6716-4958-82f5-fe4c3340090b" (UID: "c4bf6832-6716-4958-82f5-fe4c3340090b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:01:50.408331 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.408302 5015 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-provider-credentials\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:50.408331 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.408329 5015 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-provider-credentials\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:50.408331 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.408338 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bf6832-6716-4958-82f5-fe4c3340090b-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:50.408581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.408348 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf9wz\" (UniqueName: \"kubernetes.io/projected/c4bf6832-6716-4958-82f5-fe4c3340090b-kube-api-access-qf9wz\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:50.408581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.408358 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dk7k\" (UniqueName: \"kubernetes.io/projected/7b310f2a-9ee6-4550-a5f6-6edd579070b1-kube-api-access-6dk7k\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:50.812194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.812160 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg"] Nov 03 09:01:50.812758 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.812740 5015 cpu_manager.go:414] "containerMap: removing container" podUID="981df4b1-4b75-49e8-9df1-b74be2be6131" containerName="revoke-keys" Nov 03 09:01:50.812758 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.812756 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981df4b1-4b75-49e8-9df1-b74be2be6131" containerName="revoke-keys" Nov 03 09:01:50.812914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.812769 5015 cpu_manager.go:414] "containerMap: removing container" podUID="7b310f2a-9ee6-4550-a5f6-6edd579070b1" containerName="payload-processing" Nov 03 09:01:50.812914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.812775 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b310f2a-9ee6-4550-a5f6-6edd579070b1" containerName="payload-processing" Nov 03 09:01:50.812914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.812799 5015 cpu_manager.go:414] "containerMap: removing container" podUID="c4bf6832-6716-4958-82f5-fe4c3340090b" containerName="payload-processing" Nov 03 09:01:50.812914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.812804 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4bf6832-6716-4958-82f5-fe4c3340090b" containerName="payload-processing" Nov 03 09:01:50.812914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.812896 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b310f2a-9ee6-4550-a5f6-6edd579070b1" containerName="payload-processing" Nov 03 09:01:50.812914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.812911 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="981df4b1-4b75-49e8-9df1-b74be2be6131" containerName="revoke-keys" Nov 03 09:01:50.812914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.812917 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4bf6832-6716-4958-82f5-fe4c3340090b" containerName="payload-processing" Nov 03 09:01:50.816943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.816917 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:50.881530 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.878550 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg"] Nov 03 09:01:50.907378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.907342 5015 generic.go:356] "Generic (PLEG): container finished" podID="7b310f2a-9ee6-4550-a5f6-6edd579070b1" containerID="53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a" exitCode=0 Nov 03 09:01:50.907588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.907405 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" Nov 03 09:01:50.907588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.907429 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" event={"ID":"7b310f2a-9ee6-4550-a5f6-6edd579070b1","Type":"ContainerDied","Data":"53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a"} Nov 03 09:01:50.907588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.907528 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq" event={"ID":"7b310f2a-9ee6-4550-a5f6-6edd579070b1","Type":"ContainerDied","Data":"7bf6f0763a2ffc8bd9927c91c9bae3d4ba7f4ea2ca32aac9e9ccc4e191fa5fe2"} Nov 03 09:01:50.907588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.907562 5015 scope.go:122] "RemoveContainer" containerID="53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a" Nov 03 09:01:50.909811 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.909782 5015 generic.go:356] "Generic (PLEG): container finished" podID="c4bf6832-6716-4958-82f5-fe4c3340090b" containerID="4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c" exitCode=0 Nov 03 09:01:50.909907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.909841 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" event={"ID":"c4bf6832-6716-4958-82f5-fe4c3340090b","Type":"ContainerDied","Data":"4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c"} Nov 03 09:01:50.909907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.909860 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" Nov 03 09:01:50.910004 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.909922 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf" event={"ID":"c4bf6832-6716-4958-82f5-fe4c3340090b","Type":"ContainerDied","Data":"7f13ee83a77709d5f9d3212e57d3f7822f15dcb845c30ce707327233f468698b"} Nov 03 09:01:50.915707 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.915679 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:50.916244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.915757 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b52cce-6255-4264-af07-e18a4352eb5a-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:50.916244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.915804 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:50.916244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.915852 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:50.916244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.915875 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:50.916244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.916230 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:50.917192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.916263 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:50.917192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.916331 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:50.917192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.916364 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2h2l\" (UniqueName: \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-kube-api-access-l2h2l\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:50.929961 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.929927 5015 scope.go:122] "RemoveContainer" containerID="53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a" Nov 03 09:01:50.934381 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:01:50.934354 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a\": container with ID starting with 53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a not found: ID does not exist" containerID="53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a" Nov 03 09:01:50.934551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.934391 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a"} err="failed to get container status \"53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a\": rpc error: code = NotFound desc = could not find container \"53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a\": container with ID starting with 53132416304ec795457c1c589073a367b004e72566c2a2abf35f480e92a1862a not found: ID does not exist" Nov 03 09:01:50.934551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.934409 5015 scope.go:122] "RemoveContainer" containerID="4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c" Nov 03 09:01:50.970923 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.970839 5015 scope.go:122] "RemoveContainer" containerID="4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c" Nov 03 09:01:50.971225 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:01:50.971194 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c\": container with ID starting with 4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c not found: ID does not exist" containerID="4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c" Nov 03 09:01:50.971305 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:50.971238 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c"} err="failed to get container status \"4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c\": rpc error: code = NotFound desc = could not find container \"4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c\": container with ID starting with 4ef21a624c335614c16dd46e482edcaff6521072d3a507712fe45b5d4f34681c not found: ID does not exist" Nov 03 09:01:51.002729 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:01:51.002700 5015 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bf6832_6716_4958_82f5_fe4c3340090b.slice/crio-7f13ee83a77709d5f9d3212e57d3f7822f15dcb845c30ce707327233f468698b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bf6832_6716_4958_82f5_fe4c3340090b.slice\": RecentStats: unable to find data in memory cache]" Nov 03 09:01:51.017273 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.017246 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.017391 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.017309 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b52cce-6255-4264-af07-e18a4352eb5a-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.017391 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.017354 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.017578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.017404 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.017578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.017430 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.017578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.017506 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.017578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.017536 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.017578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.017567 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.017894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.017598 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2h2l\" (UniqueName: \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-kube-api-access-l2h2l\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.017894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.017877 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.017995 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.017895 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.018162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.018139 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.018761 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.018733 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.019510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.019467 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b52cce-6255-4264-af07-e18a4352eb5a-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.020258 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.020011 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.020801 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.020780 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.080716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.080646 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.082321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.082292 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2h2l\" (UniqueName: \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-kube-api-access-l2h2l\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.131270 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.131245 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:51.411515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.411403 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq"] Nov 03 09:01:51.416359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.416331 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6845cdf9f-6sxqq"] Nov 03 09:01:51.530604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.530570 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf"] Nov 03 09:01:51.534940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.534909 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-665b9fd955-8zqxf"] Nov 03 09:01:51.571533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.571503 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg"] Nov 03 09:01:51.572725 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:01:51.572700 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b52cce_6255_4264_af07_e18a4352eb5a.slice/crio-70161227d5193c791d27ece685f994ed6800052f6bce0b7ad32014a3cfc0bacb WatchSource:0}: Error finding container 70161227d5193c791d27ece685f994ed6800052f6bce0b7ad32014a3cfc0bacb: Status 404 returned error can't find the container with id 70161227d5193c791d27ece685f994ed6800052f6bce0b7ad32014a3cfc0bacb Nov 03 09:01:51.575775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.575743 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:01:51.575885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.575809 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:01:51.575885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.575843 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:01:51.923691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.923644 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" event={"ID":"e3b52cce-6255-4264-af07-e18a4352eb5a","Type":"ContainerStarted","Data":"336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41"} Nov 03 09:01:51.923887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:51.923706 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" event={"ID":"e3b52cce-6255-4264-af07-e18a4352eb5a","Type":"ContainerStarted","Data":"70161227d5193c791d27ece685f994ed6800052f6bce0b7ad32014a3cfc0bacb"} Nov 03 09:01:52.047338 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:52.047288 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" podStartSLOduration=2.047272789 podStartE2EDuration="2.047272789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:52.046517479 +0000 UTC m=+1344.971459717" watchObservedRunningTime="2026-11-03 09:01:52.047272789 +0000 UTC m=+1344.972215005" Nov 03 09:01:52.131780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:52.131748 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:52.133560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:52.133534 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 09:01:52.133641 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:52.133597 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 09:01:53.132375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:53.132339 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 09:01:53.132820 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:53.132408 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 09:01:53.206272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:53.206239 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b310f2a-9ee6-4550-a5f6-6edd579070b1" path="/var/lib/kubelet/pods/7b310f2a-9ee6-4550-a5f6-6edd579070b1/volumes" Nov 03 09:01:53.206723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:53.206709 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4bf6832-6716-4958-82f5-fe4c3340090b" path="/var/lib/kubelet/pods/c4bf6832-6716-4958-82f5-fe4c3340090b/volumes" Nov 03 09:01:54.132207 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.132172 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 09:01:54.132362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.132234 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 09:01:54.202136 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.202107 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg"] Nov 03 09:01:54.207425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.207405 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:54.268104 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.268071 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg"] Nov 03 09:01:54.293597 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.293568 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c355d1-49vkg\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:54.293873 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.293845 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c355d1-49vkg\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:54.293978 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.293957 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkpfd\" (UniqueName: \"kubernetes.io/projected/2f85603a-e596-4a81-a6e9-d843f4be7901-kube-api-access-nkpfd\") pod \"maas-api-revoke-sub-d8c355d1-49vkg\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:54.312656 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.312630 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 09:01:54.312812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.312798 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:01:54.394866 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.394784 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c355d1-49vkg\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:54.395027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.394906 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c355d1-49vkg\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:54.395027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.394945 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkpfd\" (UniqueName: \"kubernetes.io/projected/2f85603a-e596-4a81-a6e9-d843f4be7901-kube-api-access-nkpfd\") pod \"maas-api-revoke-sub-d8c355d1-49vkg\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:54.395501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.395449 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c355d1-49vkg\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:54.397554 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.397529 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c355d1-49vkg\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:54.462274 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.462236 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkpfd\" (UniqueName: \"kubernetes.io/projected/2f85603a-e596-4a81-a6e9-d843f4be7901-kube-api-access-nkpfd\") pod \"maas-api-revoke-sub-d8c355d1-49vkg\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:54.521604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.521574 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:54.936057 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.936030 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg"] Nov 03 09:01:54.979121 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:54.979085 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" event={"ID":"2f85603a-e596-4a81-a6e9-d843f4be7901","Type":"ContainerStarted","Data":"12ff6e25482026c157a11a9a7900502c244b6e390dacf66df14c9cd71b422093"} Nov 03 09:01:55.131971 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:55.131935 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 09:01:55.132147 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:55.131990 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 09:01:55.987329 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:55.987289 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" event={"ID":"2f85603a-e596-4a81-a6e9-d843f4be7901","Type":"ContainerStarted","Data":"def7213e5ccb260d00f8ea660e675a3ff1edff1541fbfc9ff8d0c0e453a59486"} Nov 03 09:01:56.131068 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:56.131007 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" podStartSLOduration=2.130991933 podStartE2EDuration="2.130991933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:56.12860665 +0000 UTC m=+1349.053548899" watchObservedRunningTime="2026-11-03 09:01:56.130991933 +0000 UTC m=+1349.055934149" Nov 03 09:01:56.132056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:56.132029 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 09:01:56.132197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:56.132080 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 09:01:56.996712 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:56.996677 5015 generic.go:356] "Generic (PLEG): container finished" podID="2f85603a-e596-4a81-a6e9-d843f4be7901" containerID="def7213e5ccb260d00f8ea660e675a3ff1edff1541fbfc9ff8d0c0e453a59486" exitCode=0 Nov 03 09:01:56.997111 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:56.996732 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" event={"ID":"2f85603a-e596-4a81-a6e9-d843f4be7901","Type":"ContainerDied","Data":"def7213e5ccb260d00f8ea660e675a3ff1edff1541fbfc9ff8d0c0e453a59486"} Nov 03 09:01:57.132050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:57.132014 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 09:01:57.132245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:57.132076 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 09:01:58.132195 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.132157 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 09:01:58.132699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.132235 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 09:01:58.177845 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.177821 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:58.245325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.245296 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f85603a-e596-4a81-a6e9-d843f4be7901-kube-api-access-nkpfd\" (UniqueName: \"kubernetes.io/projected/2f85603a-e596-4a81-a6e9-d843f4be7901-kube-api-access-nkpfd\") pod \"2f85603a-e596-4a81-a6e9-d843f4be7901\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " Nov 03 09:01:58.245458 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.245375 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-openshift-ca\") pod \"2f85603a-e596-4a81-a6e9-d843f4be7901\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " Nov 03 09:01:58.245458 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.245421 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-xks-ca\") pod \"2f85603a-e596-4a81-a6e9-d843f4be7901\" (UID: \"2f85603a-e596-4a81-a6e9-d843f4be7901\") " Nov 03 09:01:58.245818 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.245789 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-openshift-ca" pod "2f85603a-e596-4a81-a6e9-d843f4be7901" (UID: "2f85603a-e596-4a81-a6e9-d843f4be7901"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:01:58.246221 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.246161 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:58.247677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.247655 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-xks-ca" pod "2f85603a-e596-4a81-a6e9-d843f4be7901" (UID: "2f85603a-e596-4a81-a6e9-d843f4be7901"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:01:58.247774 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.247675 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f85603a-e596-4a81-a6e9-d843f4be7901-kube-api-access-nkpfd" pod "2f85603a-e596-4a81-a6e9-d843f4be7901" (UID: "2f85603a-e596-4a81-a6e9-d843f4be7901"). InnerVolumeSpecName "kube-api-access-nkpfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:01:58.347560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.347524 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f85603a-e596-4a81-a6e9-d843f4be7901-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:58.347560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:58.347553 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkpfd\" (UniqueName: \"kubernetes.io/projected/2f85603a-e596-4a81-a6e9-d843f4be7901-kube-api-access-nkpfd\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:01:59.014547 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:59.014509 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" event={"ID":"2f85603a-e596-4a81-a6e9-d843f4be7901","Type":"ContainerDied","Data":"12ff6e25482026c157a11a9a7900502c244b6e390dacf66df14c9cd71b422093"} Nov 03 09:01:59.014838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:59.014563 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg" Nov 03 09:01:59.014838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:59.014572 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12ff6e25482026c157a11a9a7900502c244b6e390dacf66df14c9cd71b422093" Nov 03 09:01:59.136375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:59.136349 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:59.136898 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:59.136878 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:01:59.137634 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:01:59.137612 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:02:07.805503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:07.804638 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:07.975505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:07.972994 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:08.004503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:08.004010 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:08.017989 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:08.015601 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podUID="21ad0cfe-e3c6-4770-b0b3-7c512e7f1830" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 09:02:08.405513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:08.404612 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:08.576505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:08.507051 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:08.666558 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:08.665563 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:08.812167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:08.812123 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:08.813097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:08.813075 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:02:08.988502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:08.980270 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:08.988502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:08.980706 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 09:02:09.039507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.032574 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podUID="21ad0cfe-e3c6-4770-b0b3-7c512e7f1830" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:09.039507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.033140 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 09:02:09.064504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.057832 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:09.064504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.058419 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 09:02:09.252557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.238032 5015 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:09.478504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.474688 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 09:02:09.626562 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.599790 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:09.654607 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.604200 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 09:02:09.654607 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.650087 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 09:02:09.759519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.704610 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:09.759519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.705796 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 09:02:09.898188 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.898088 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Nov 03 09:02:09.933392 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.933361 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 09:02:09.939967 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.939944 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 09:02:09.950869 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.950846 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:02:09.984503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.984458 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 09:02:09.986288 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:09.986256 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 03 09:02:10.036551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:10.036460 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podUID="21ad0cfe-e3c6-4770-b0b3-7c512e7f1830" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:10.180565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:10.176444 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 09:02:10.228757 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:10.228721 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 09:02:10.228959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:10.228787 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 09:02:12.339328 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.339297 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8"] Nov 03 09:02:12.340246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.340221 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2f85603a-e596-4a81-a6e9-d843f4be7901" containerName="revoke-keys" Nov 03 09:02:12.340246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.340245 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f85603a-e596-4a81-a6e9-d843f4be7901" containerName="revoke-keys" Nov 03 09:02:12.340437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.340389 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f85603a-e596-4a81-a6e9-d843f4be7901" containerName="revoke-keys" Nov 03 09:02:12.344362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.344340 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.353068 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.353018 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.353180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.353167 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.353248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.353209 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.353248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.353232 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.353392 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.353371 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.353459 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.353401 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.353568 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.353547 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggln6\" (UniqueName: \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-kube-api-access-ggln6\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.353637 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.353584 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.353637 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.353614 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.389823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.389799 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c"] Nov 03 09:02:12.393941 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.393919 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.442667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.442626 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8"] Nov 03 09:02:12.455060 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455020 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.455230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455097 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.455230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455143 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.455230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455180 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.455230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455220 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.455454 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455267 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.455454 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455291 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd5pj\" (UniqueName: \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-kube-api-access-wd5pj\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.455454 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455325 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.455454 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455375 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.455454 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455437 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.455750 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455468 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.455750 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455546 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.455750 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455595 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggln6\" (UniqueName: \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-kube-api-access-ggln6\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.455750 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455632 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.455750 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455661 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.455750 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455746 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.456049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455812 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.456049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.455836 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.456405 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.456380 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.458874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.458373 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.458874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.458657 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.459359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.459338 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.459753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.459700 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.462020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.461997 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c"] Nov 03 09:02:12.462853 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.462828 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.464073 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.464051 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.516798 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.516760 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.516943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.516843 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggln6\" (UniqueName: \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-kube-api-access-ggln6\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.557128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557097 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.557128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557130 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.557362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557166 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.557362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557199 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.557362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557222 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.557362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557256 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd5pj\" (UniqueName: \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-kube-api-access-wd5pj\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.557362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557287 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.557362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557334 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.557747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557409 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.557747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557581 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.557747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557657 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.557905 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.557855 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.558062 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.558038 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.558537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.558512 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.559769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.559745 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.560044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.560027 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.616143 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.616074 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.616318 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.616299 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd5pj\" (UniqueName: \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-kube-api-access-wd5pj\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.658590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.658555 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:12.707778 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.707744 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:12.994579 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:12.994549 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz"] Nov 03 09:02:13.000449 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.000424 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:13.054448 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.054415 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz"] Nov 03 09:02:13.055118 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.055095 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 09:02:13.055216 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.055137 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:02:13.063012 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.062984 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49d332e-tdgfz\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:13.063156 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.063094 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49d332e-tdgfz\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:13.063368 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.063350 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v42v4\" (UniqueName: \"kubernetes.io/projected/2a1e693b-9d3e-4082-91e9-ee5961cadc17-kube-api-access-v42v4\") pod \"maas-api-revoke-sub-f49d332e-tdgfz\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:13.096469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.096445 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8"] Nov 03 09:02:13.098164 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:02:13.098135 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7cc2ee_3c0b_48fd_9701_87b4c692aa9d.slice/crio-f6ad44e74cebb1d5198f331fd5154cbf70ae4fb1129225982c2ea615541f37d0 WatchSource:0}: Error finding container f6ad44e74cebb1d5198f331fd5154cbf70ae4fb1129225982c2ea615541f37d0: Status 404 returned error can't find the container with id f6ad44e74cebb1d5198f331fd5154cbf70ae4fb1129225982c2ea615541f37d0 Nov 03 09:02:13.100724 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.100690 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:13.100797 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.100758 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:13.100831 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.100796 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:13.151924 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.151898 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c"] Nov 03 09:02:13.167444 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.164522 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42v4\" (UniqueName: \"kubernetes.io/projected/2a1e693b-9d3e-4082-91e9-ee5961cadc17-kube-api-access-v42v4\") pod \"maas-api-revoke-sub-f49d332e-tdgfz\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:13.167444 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.164618 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49d332e-tdgfz\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:13.167444 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.164696 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49d332e-tdgfz\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:13.167444 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.167395 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49d332e-tdgfz\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:13.184259 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.184206 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49d332e-tdgfz\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:13.187737 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.187707 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:13.187808 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.187779 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:13.187844 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.187815 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:13.226417 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.226386 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42v4\" (UniqueName: \"kubernetes.io/projected/2a1e693b-9d3e-4082-91e9-ee5961cadc17-kube-api-access-v42v4\") pod \"maas-api-revoke-sub-f49d332e-tdgfz\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:13.314719 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.314680 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:13.712968 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.712747 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz"] Nov 03 09:02:13.719500 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:02:13.714895 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1e693b_9d3e_4082_91e9_ee5961cadc17.slice/crio-a41bade506836b133ac0670cc1f1eb9e16aa7f9d978cf81a55eae5baa3d26dc4 WatchSource:0}: Error finding container a41bade506836b133ac0670cc1f1eb9e16aa7f9d978cf81a55eae5baa3d26dc4: Status 404 returned error can't find the container with id a41bade506836b133ac0670cc1f1eb9e16aa7f9d978cf81a55eae5baa3d26dc4 Nov 03 09:02:13.809362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.809328 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" event={"ID":"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d","Type":"ContainerStarted","Data":"a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a"} Nov 03 09:02:13.809539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.809394 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" event={"ID":"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d","Type":"ContainerStarted","Data":"f6ad44e74cebb1d5198f331fd5154cbf70ae4fb1129225982c2ea615541f37d0"} Nov 03 09:02:13.810773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.810750 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" event={"ID":"2a1e693b-9d3e-4082-91e9-ee5961cadc17","Type":"ContainerStarted","Data":"a41bade506836b133ac0670cc1f1eb9e16aa7f9d978cf81a55eae5baa3d26dc4"} Nov 03 09:02:13.814525 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.812439 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" event={"ID":"1ffd055e-1668-4ca2-8495-6fe0a3cc5923","Type":"ContainerStarted","Data":"d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3"} Nov 03 09:02:13.814525 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.812504 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" event={"ID":"1ffd055e-1668-4ca2-8495-6fe0a3cc5923","Type":"ContainerStarted","Data":"8fb2978ad088b1d026d02f9464349b9e04caeda72c8024c8c469b4bb0ff26667"} Nov 03 09:02:13.957502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:13.955291 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" podStartSLOduration=1.955267805 podStartE2EDuration="1.955267805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:13.934380185 +0000 UTC m=+1366.859322434" watchObservedRunningTime="2026-11-03 09:02:13.955267805 +0000 UTC m=+1366.880210028" Nov 03 09:02:14.092502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:14.077196 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" podStartSLOduration=2.077174228 podStartE2EDuration="2.077174228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:14.066892148 +0000 UTC m=+1366.991834366" watchObservedRunningTime="2026-11-03 09:02:14.077174228 +0000 UTC m=+1367.002116446" Nov 03 09:02:14.661953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:14.661918 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:14.667975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:14.667608 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 09:02:14.667975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:14.667671 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 09:02:14.708177 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:14.708143 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:14.709240 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:14.709126 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 09:02:14.709240 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:14.709180 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 09:02:14.825790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:14.825523 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" event={"ID":"2a1e693b-9d3e-4082-91e9-ee5961cadc17","Type":"ContainerStarted","Data":"299bdfbf871469deadcc3c30d938436ea745ebfb19955d5356d48697d6987cfc"} Nov 03 09:02:15.008520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:15.008220 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" podStartSLOduration=3.008200601 podStartE2EDuration="3.008200601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:14.988196364 +0000 UTC m=+1367.913138606" watchObservedRunningTime="2026-11-03 09:02:15.008200601 +0000 UTC m=+1367.933142819" Nov 03 09:02:15.666503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:15.664843 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 09:02:15.666503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:15.664906 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 09:02:15.709516 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:15.709056 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 09:02:15.709516 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:15.709120 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 09:02:15.832839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:15.832801 5015 generic.go:356] "Generic (PLEG): container finished" podID="2a1e693b-9d3e-4082-91e9-ee5961cadc17" containerID="299bdfbf871469deadcc3c30d938436ea745ebfb19955d5356d48697d6987cfc" exitCode=0 Nov 03 09:02:15.833286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:15.832884 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" event={"ID":"2a1e693b-9d3e-4082-91e9-ee5961cadc17","Type":"ContainerDied","Data":"299bdfbf871469deadcc3c30d938436ea745ebfb19955d5356d48697d6987cfc"} Nov 03 09:02:16.666355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:16.665885 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 09:02:16.666355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:16.665949 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 09:02:16.709240 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:16.709131 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 09:02:16.709240 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:16.709195 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 09:02:16.804834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:16.804795 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2"] Nov 03 09:02:16.805324 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:16.805294 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" containerID="cri-o://48bc24f1d4e4bd7fbd01bf538b32218a0603fabd534f1589ae57560a605d39be" gracePeriod=30 Nov 03 09:02:17.019054 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:02:17.019015 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:02:17.314213 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.314179 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:17.415064 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.415013 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-openshift-ca\") pod \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " Nov 03 09:02:17.415235 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.415125 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1e693b-9d3e-4082-91e9-ee5961cadc17-kube-api-access-v42v4\" (UniqueName: \"kubernetes.io/projected/2a1e693b-9d3e-4082-91e9-ee5961cadc17-kube-api-access-v42v4\") pod \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " Nov 03 09:02:17.415317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.415236 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-xks-ca\") pod \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\" (UID: \"2a1e693b-9d3e-4082-91e9-ee5961cadc17\") " Nov 03 09:02:17.415749 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.415718 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-openshift-ca" pod "2a1e693b-9d3e-4082-91e9-ee5961cadc17" (UID: "2a1e693b-9d3e-4082-91e9-ee5961cadc17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:02:17.417888 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.417864 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-xks-ca" pod "2a1e693b-9d3e-4082-91e9-ee5961cadc17" (UID: "2a1e693b-9d3e-4082-91e9-ee5961cadc17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:02:17.418141 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.418119 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1e693b-9d3e-4082-91e9-ee5961cadc17-kube-api-access-v42v4" pod "2a1e693b-9d3e-4082-91e9-ee5961cadc17" (UID: "2a1e693b-9d3e-4082-91e9-ee5961cadc17"). InnerVolumeSpecName "kube-api-access-v42v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:02:17.516565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.516453 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v42v4\" (UniqueName: \"kubernetes.io/projected/2a1e693b-9d3e-4082-91e9-ee5961cadc17-kube-api-access-v42v4\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:17.516565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.516498 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:17.516565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.516508 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e693b-9d3e-4082-91e9-ee5961cadc17-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:17.664502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.664019 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 09:02:17.664502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.664081 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 09:02:17.708912 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.708876 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 09:02:17.709088 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.708928 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 09:02:17.927436 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.927355 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" Nov 03 09:02:17.927436 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.927375 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz" event={"ID":"2a1e693b-9d3e-4082-91e9-ee5961cadc17","Type":"ContainerDied","Data":"a41bade506836b133ac0670cc1f1eb9e16aa7f9d978cf81a55eae5baa3d26dc4"} Nov 03 09:02:17.927636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:17.927444 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a41bade506836b133ac0670cc1f1eb9e16aa7f9d978cf81a55eae5baa3d26dc4" Nov 03 09:02:18.670505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:18.659885 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 09:02:18.670505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:18.659964 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 09:02:18.708755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:18.708716 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 09:02:18.708905 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:18.708779 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 09:02:19.659109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:19.659066 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 09:02:19.659296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:19.659143 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 09:02:19.708429 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:19.708396 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 09:02:19.708842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:19.708461 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 09:02:20.663509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:20.663461 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:20.663958 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:20.663946 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:20.664733 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:20.664710 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:02:20.712449 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:20.712425 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:20.712834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:20.712692 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:20.713467 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:20.713445 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:02:23.050521 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.046546 5015 generic.go:356] "Generic (PLEG): container finished" podID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerID="48bc24f1d4e4bd7fbd01bf538b32218a0603fabd534f1589ae57560a605d39be" exitCode=0 Nov 03 09:02:23.050521 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.046775 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" event={"ID":"5ecc5d8b-4c77-448e-afd9-795bf6833b69","Type":"ContainerDied","Data":"48bc24f1d4e4bd7fbd01bf538b32218a0603fabd534f1589ae57560a605d39be"} Nov 03 09:02:23.050521 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.046835 5015 scope.go:122] "RemoveContainer" containerID="5b99d6605bd39957869b5ec5ff1fda856aed0f4ef0d6a7920dead0275268ce99" Nov 03 09:02:23.386622 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.367585 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:23.705553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.692879 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 09:02:23.793353 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.793320 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b"] Nov 03 09:02:23.794536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.794510 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2a1e693b-9d3e-4082-91e9-ee5961cadc17" containerName="revoke-keys" Nov 03 09:02:23.794686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.794672 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1e693b-9d3e-4082-91e9-ee5961cadc17" containerName="revoke-keys" Nov 03 09:02:23.794788 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.794777 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" Nov 03 09:02:23.794879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.794867 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" Nov 03 09:02:23.794968 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.794957 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" Nov 03 09:02:23.795048 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.795038 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" Nov 03 09:02:23.795299 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.795282 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1e693b-9d3e-4082-91e9-ee5961cadc17" containerName="revoke-keys" Nov 03 09:02:23.795409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.795397 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" Nov 03 09:02:23.799555 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.799537 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" containerName="istio-proxy" Nov 03 09:02:23.801043 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.801017 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-kube-api-access-cl46c\" (UniqueName: \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-kube-api-access-cl46c\") pod \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " Nov 03 09:02:23.801671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.801650 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-credential-socket\") pod \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " Nov 03 09:02:23.802157 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.802140 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-data\") pod \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " Nov 03 09:02:23.802583 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.802563 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-certs\") pod \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " Nov 03 09:02:23.802997 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.802979 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istiod-ca-cert\") pod \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " Nov 03 09:02:23.805675 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.805641 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-socket\") pod \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " Nov 03 09:02:23.805882 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.805866 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-envoy\") pod \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " Nov 03 09:02:23.806061 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.806033 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-token\" (UniqueName: \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-token\") pod \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " Nov 03 09:02:23.807279 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.806749 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-podinfo\") pod \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\" (UID: \"5ecc5d8b-4c77-448e-afd9-795bf6833b69\") " Nov 03 09:02:23.807375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.802041 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-credential-socket" pod "5ecc5d8b-4c77-448e-afd9-795bf6833b69" (UID: "5ecc5d8b-4c77-448e-afd9-795bf6833b69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:02:23.807375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.802455 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-data" pod "5ecc5d8b-4c77-448e-afd9-795bf6833b69" (UID: "5ecc5d8b-4c77-448e-afd9-795bf6833b69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:02:23.807375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.802881 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-certs" pod "5ecc5d8b-4c77-448e-afd9-795bf6833b69" (UID: "5ecc5d8b-4c77-448e-afd9-795bf6833b69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:02:23.807375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.806687 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-socket" pod "5ecc5d8b-4c77-448e-afd9-795bf6833b69" (UID: "5ecc5d8b-4c77-448e-afd9-795bf6833b69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:02:23.808409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.808381 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:23.808562 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.808545 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:23.808703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.808687 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:23.808833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.808818 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:23.809150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.809130 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istiod-ca-cert" pod "5ecc5d8b-4c77-448e-afd9-795bf6833b69" (UID: "5ecc5d8b-4c77-448e-afd9-795bf6833b69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:02:23.818450 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.818430 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b"] Nov 03 09:02:23.818749 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.818680 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:23.907099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.906818 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-envoy" pod "5ecc5d8b-4c77-448e-afd9-795bf6833b69" (UID: "5ecc5d8b-4c77-448e-afd9-795bf6833b69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:02:23.907099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.906945 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-kube-api-access-cl46c" pod "5ecc5d8b-4c77-448e-afd9-795bf6833b69" (UID: "5ecc5d8b-4c77-448e-afd9-795bf6833b69"). InnerVolumeSpecName "kube-api-access-cl46c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:02:23.907099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.907006 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-podinfo" pod "5ecc5d8b-4c77-448e-afd9-795bf6833b69" (UID: "5ecc5d8b-4c77-448e-afd9-795bf6833b69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:02:23.907099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.907062 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-token" pod "5ecc5d8b-4c77-448e-afd9-795bf6833b69" (UID: "5ecc5d8b-4c77-448e-afd9-795bf6833b69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:02:23.916545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.914431 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6b4aee-5xj5b\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:23.916545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.914652 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69nbl\" (UniqueName: \"kubernetes.io/projected/651dfc4a-7ddd-49e9-ae18-386a259022ac-kube-api-access-69nbl\") pod \"maas-api-revoke-sub-cd6b4aee-5xj5b\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:23.916545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.914710 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6b4aee-5xj5b\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:23.916545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.914831 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl46c\" (UniqueName: \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-kube-api-access-cl46c\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:23.916545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.914849 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:23.916545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.914865 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:23.916545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.914879 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:23.916545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.914893 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecc5d8b-4c77-448e-afd9-795bf6833b69-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:23.967489 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.954098 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 09:02:23.967489 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.954279 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:02:23.967489 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.958567 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:23.967489 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.958873 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 09:02:23.967489 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:23.959151 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 09:02:24.051547 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.033333 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6b4aee-5xj5b\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:24.051547 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.033567 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69nbl\" (UniqueName: \"kubernetes.io/projected/651dfc4a-7ddd-49e9-ae18-386a259022ac-kube-api-access-69nbl\") pod \"maas-api-revoke-sub-cd6b4aee-5xj5b\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:24.051547 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.033634 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6b4aee-5xj5b\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:24.051547 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.034533 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6b4aee-5xj5b\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:24.051547 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.039545 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 03 09:02:24.060956 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.060923 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6b4aee-5xj5b\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:24.196724 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.196685 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" event={"ID":"5ecc5d8b-4c77-448e-afd9-795bf6833b69","Type":"ContainerDied","Data":"8f248397b5e05060415c6bd21c6f29b6da8e7e5af2968e07105ba3e4db206d7b"} Nov 03 09:02:24.197090 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.197072 5015 scope.go:122] "RemoveContainer" containerID="48bc24f1d4e4bd7fbd01bf538b32218a0603fabd534f1589ae57560a605d39be" Nov 03 09:02:24.198942 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.197268 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2" Nov 03 09:02:24.239041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.238743 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69nbl\" (UniqueName: \"kubernetes.io/projected/651dfc4a-7ddd-49e9-ae18-386a259022ac-kube-api-access-69nbl\") pod \"maas-api-revoke-sub-cd6b4aee-5xj5b\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:24.432030 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.431998 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2"] Nov 03 09:02:24.438186 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.438148 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-kqwn2"] Nov 03 09:02:24.459358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.459328 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:24.859863 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:02:24.859834 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651dfc4a_7ddd_49e9_ae18_386a259022ac.slice/crio-62e84a60dc091e9449973076ebef1d7e6407ca881ec12a15d73763f72209a003 WatchSource:0}: Error finding container 62e84a60dc091e9449973076ebef1d7e6407ca881ec12a15d73763f72209a003: Status 404 returned error can't find the container with id 62e84a60dc091e9449973076ebef1d7e6407ca881ec12a15d73763f72209a003 Nov 03 09:02:24.861358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:24.861329 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b"] Nov 03 09:02:25.205341 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:25.205310 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ecc5d8b-4c77-448e-afd9-795bf6833b69" path="/var/lib/kubelet/pods/5ecc5d8b-4c77-448e-afd9-795bf6833b69/volumes" Nov 03 09:02:25.206134 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:25.206103 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" event={"ID":"651dfc4a-7ddd-49e9-ae18-386a259022ac","Type":"ContainerStarted","Data":"83c0df5071456ad501f7e68c381bcd736e39c1fc0b6f3e5ca392c8bddde0f20f"} Nov 03 09:02:25.206306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:25.206163 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" event={"ID":"651dfc4a-7ddd-49e9-ae18-386a259022ac","Type":"ContainerStarted","Data":"62e84a60dc091e9449973076ebef1d7e6407ca881ec12a15d73763f72209a003"} Nov 03 09:02:25.331708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:25.331648 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" podStartSLOduration=3.331634581 podStartE2EDuration="3.331634581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:25.331391759 +0000 UTC m=+1378.256333996" watchObservedRunningTime="2026-11-03 09:02:25.331634581 +0000 UTC m=+1378.256576830" Nov 03 09:02:26.216776 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:26.216690 5015 generic.go:356] "Generic (PLEG): container finished" podID="651dfc4a-7ddd-49e9-ae18-386a259022ac" containerID="83c0df5071456ad501f7e68c381bcd736e39c1fc0b6f3e5ca392c8bddde0f20f" exitCode=0 Nov 03 09:02:26.217225 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:26.216772 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" event={"ID":"651dfc4a-7ddd-49e9-ae18-386a259022ac","Type":"ContainerDied","Data":"83c0df5071456ad501f7e68c381bcd736e39c1fc0b6f3e5ca392c8bddde0f20f"} Nov 03 09:02:27.390505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:27.390454 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:27.444970 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:27.444939 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-xks-ca\") pod \"651dfc4a-7ddd-49e9-ae18-386a259022ac\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " Nov 03 09:02:27.445118 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:27.444994 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651dfc4a-7ddd-49e9-ae18-386a259022ac-kube-api-access-69nbl\" (UniqueName: \"kubernetes.io/projected/651dfc4a-7ddd-49e9-ae18-386a259022ac-kube-api-access-69nbl\") pod \"651dfc4a-7ddd-49e9-ae18-386a259022ac\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " Nov 03 09:02:27.445177 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:27.445162 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-openshift-ca\") pod \"651dfc4a-7ddd-49e9-ae18-386a259022ac\" (UID: \"651dfc4a-7ddd-49e9-ae18-386a259022ac\") " Nov 03 09:02:27.445589 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:27.445555 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-openshift-ca" pod "651dfc4a-7ddd-49e9-ae18-386a259022ac" (UID: "651dfc4a-7ddd-49e9-ae18-386a259022ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:02:27.445880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:27.445831 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:27.447290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:27.447264 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-xks-ca" pod "651dfc4a-7ddd-49e9-ae18-386a259022ac" (UID: "651dfc4a-7ddd-49e9-ae18-386a259022ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:02:27.447290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:27.447274 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651dfc4a-7ddd-49e9-ae18-386a259022ac-kube-api-access-69nbl" pod "651dfc4a-7ddd-49e9-ae18-386a259022ac" (UID: "651dfc4a-7ddd-49e9-ae18-386a259022ac"). InnerVolumeSpecName "kube-api-access-69nbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:02:27.526856 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:27.526776 5015 scope.go:122] "RemoveContainer" containerID="0e97df507cb51a34dbafd624d5f1c1e6774362a838bc7d4221dd883183b6b1ab" Nov 03 09:02:27.546844 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:27.546807 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651dfc4a-7ddd-49e9-ae18-386a259022ac-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:27.546844 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:27.546831 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69nbl\" (UniqueName: \"kubernetes.io/projected/651dfc4a-7ddd-49e9-ae18-386a259022ac-kube-api-access-69nbl\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:02:28.231631 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:28.231602 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" Nov 03 09:02:28.231631 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:28.231624 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b" event={"ID":"651dfc4a-7ddd-49e9-ae18-386a259022ac","Type":"ContainerDied","Data":"62e84a60dc091e9449973076ebef1d7e6407ca881ec12a15d73763f72209a003"} Nov 03 09:02:28.231843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:28.231668 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62e84a60dc091e9449973076ebef1d7e6407ca881ec12a15d73763f72209a003" Nov 03 09:02:37.773558 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.773187 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2"] Nov 03 09:02:37.777498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.774178 5015 cpu_manager.go:414] "containerMap: removing container" podUID="651dfc4a-7ddd-49e9-ae18-386a259022ac" containerName="revoke-keys" Nov 03 09:02:37.777498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.774203 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651dfc4a-7ddd-49e9-ae18-386a259022ac" containerName="revoke-keys" Nov 03 09:02:37.777498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.774390 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="651dfc4a-7ddd-49e9-ae18-386a259022ac" containerName="revoke-keys" Nov 03 09:02:37.785855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.785829 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:37.847502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.844837 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b0c653f4-openshift-default-dockercfg-plrtg\"" Nov 03 09:02:37.973570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.936276 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-envoy\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:37.973570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.936332 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-data\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:37.973570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.936387 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-token\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:37.973570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.936458 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-socket\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:37.973570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.936529 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-credential-socket\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:37.973570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.936576 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istiod-ca-cert\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:37.973570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.936615 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr2bx\" (UniqueName: \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-kube-api-access-nr2bx\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:37.973570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.936805 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-certs\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:37.973570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:37.936843 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-podinfo\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.060557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.044318 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-podinfo\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.060839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.060810 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-envoy\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.197569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.197535 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-data\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.197858 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.197834 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-token\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.224398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.224364 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-socket\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.225089 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.225066 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-credential-socket\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.226834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.224974 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-socket\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.226834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.198748 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-data\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.226834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.214693 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2"] Nov 03 09:02:38.226834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.225531 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-credential-socket\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.227337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.227317 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istiod-ca-cert\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.241555 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.241526 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr2bx\" (UniqueName: \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-kube-api-access-nr2bx\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.242150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.240267 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istiod-ca-cert\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.242529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.242507 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-certs\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.243046 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.242909 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-certs\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.398561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.395672 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-envoy\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.648159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.648081 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-token\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.703022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.702948 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-podinfo\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.727303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.727270 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr2bx\" (UniqueName: \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-kube-api-access-nr2bx\") pod \"e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:38.729867 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:38.729842 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:39.944504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:39.937724 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 09:02:39.944504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:39.939395 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:39.953373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:39.952019 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Nov 03 09:02:39.995259 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:39.985327 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:40.100407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:40.100369 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2"] Nov 03 09:02:40.147599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:40.147552 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:40.195248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:40.194452 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:40.195248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:40.194545 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:40.195248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:40.187505 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:40.210691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:40.210653 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podUID="21ad0cfe-e3c6-4770-b0b3-7c512e7f1830" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 09:02:40.928988 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:40.928938 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" event={"ID":"911aeeac-4f94-4ee1-a6c6-46db527d4f0f","Type":"ContainerStarted","Data":"d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339"} Nov 03 09:02:40.929183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:40.929007 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" event={"ID":"911aeeac-4f94-4ee1-a6c6-46db527d4f0f","Type":"ContainerStarted","Data":"b90fb52997fdde30426578686e29560decef175e07160d043128e48fa73cfe5a"} Nov 03 09:02:41.050971 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:41.050911 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" podStartSLOduration=4.050895326 podStartE2EDuration="4.050895326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:41.048092042 +0000 UTC m=+1393.973034288" watchObservedRunningTime="2026-11-03 09:02:41.050895326 +0000 UTC m=+1393.975837543" Nov 03 09:02:41.731576 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:41.731544 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:41.736651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:41.736625 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:41.936944 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:41.936910 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:41.938070 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:41.938042 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:02:46.909892 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:02:46.909852 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:02:47.696545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.696509 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf"] Nov 03 09:02:47.700787 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.700767 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:02:47.758283 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.758254 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf"] Nov 03 09:02:47.793999 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.793952 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nvtt\" (UniqueName: \"kubernetes.io/projected/a55ec8f5-94f6-4c33-bc9e-36314df85e35-kube-api-access-5nvtt\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf\" (UID: \"a55ec8f5-94f6-4c33-bc9e-36314df85e35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:02:47.794246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.794224 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55ec8f5-94f6-4c33-bc9e-36314df85e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf\" (UID: \"a55ec8f5-94f6-4c33-bc9e-36314df85e35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:02:47.803875 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.803847 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b0c653f4-dockercfg-z7pkm\"" Nov 03 09:02:47.804095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.803848 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b0c653f4\"" Nov 03 09:02:47.877500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.876645 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf"] Nov 03 09:02:47.903570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.881990 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:02:47.913274 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.913232 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55ec8f5-94f6-4c33-bc9e-36314df85e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf\" (UID: \"a55ec8f5-94f6-4c33-bc9e-36314df85e35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:02:47.913817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.913384 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nvtt\" (UniqueName: \"kubernetes.io/projected/a55ec8f5-94f6-4c33-bc9e-36314df85e35-kube-api-access-5nvtt\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf\" (UID: \"a55ec8f5-94f6-4c33-bc9e-36314df85e35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:02:47.914787 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.914763 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55ec8f5-94f6-4c33-bc9e-36314df85e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf\" (UID: \"a55ec8f5-94f6-4c33-bc9e-36314df85e35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:02:47.956559 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.949558 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf"] Nov 03 09:02:47.977529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:47.976434 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nvtt\" (UniqueName: \"kubernetes.io/projected/a55ec8f5-94f6-4c33-bc9e-36314df85e35-kube-api-access-5nvtt\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf\" (UID: \"a55ec8f5-94f6-4c33-bc9e-36314df85e35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:02:48.015193 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.015163 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzkl4\" (UniqueName: \"kubernetes.io/projected/24d00911-860c-454b-b18d-4bed094c4d82-kube-api-access-tzkl4\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf\" (UID: \"24d00911-860c-454b-b18d-4bed094c4d82\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:02:48.015399 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.015204 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d00911-860c-454b-b18d-4bed094c4d82-plugins-config-volume\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf\" (UID: \"24d00911-860c-454b-b18d-4bed094c4d82\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:02:48.016820 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.016793 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:02:48.117340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.116284 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkl4\" (UniqueName: \"kubernetes.io/projected/24d00911-860c-454b-b18d-4bed094c4d82-kube-api-access-tzkl4\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf\" (UID: \"24d00911-860c-454b-b18d-4bed094c4d82\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:02:48.117340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.116323 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d00911-860c-454b-b18d-4bed094c4d82-plugins-config-volume\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf\" (UID: \"24d00911-860c-454b-b18d-4bed094c4d82\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:02:48.117340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.117058 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d00911-860c-454b-b18d-4bed094c4d82-plugins-config-volume\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf\" (UID: \"24d00911-860c-454b-b18d-4bed094c4d82\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:02:48.178376 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.178333 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzkl4\" (UniqueName: \"kubernetes.io/projected/24d00911-860c-454b-b18d-4bed094c4d82-kube-api-access-tzkl4\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf\" (UID: \"24d00911-860c-454b-b18d-4bed094c4d82\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:02:48.236044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.236014 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:02:48.412141 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.412109 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf"] Nov 03 09:02:48.412357 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:02:48.412335 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55ec8f5_94f6_4c33_bc9e_36314df85e35.slice/crio-d191459941fb98afd0c0d56b06989c9598375e80b75728687c714eeeacef8d73 WatchSource:0}: Error finding container d191459941fb98afd0c0d56b06989c9598375e80b75728687c714eeeacef8d73: Status 404 returned error can't find the container with id d191459941fb98afd0c0d56b06989c9598375e80b75728687c714eeeacef8d73 Nov 03 09:02:48.619985 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:02:48.619954 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d00911_860c_454b_b18d_4bed094c4d82.slice/crio-52d2f7d0a1f30140eb0074f9431210e0541c554e95890fd148a536b8089fe6a5 WatchSource:0}: Error finding container 52d2f7d0a1f30140eb0074f9431210e0541c554e95890fd148a536b8089fe6a5: Status 404 returned error can't find the container with id 52d2f7d0a1f30140eb0074f9431210e0541c554e95890fd148a536b8089fe6a5 Nov 03 09:02:48.622314 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.622287 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf"] Nov 03 09:02:48.995882 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.995850 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" event={"ID":"a55ec8f5-94f6-4c33-bc9e-36314df85e35","Type":"ContainerStarted","Data":"0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42"} Nov 03 09:02:48.996300 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.995912 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" event={"ID":"a55ec8f5-94f6-4c33-bc9e-36314df85e35","Type":"ContainerStarted","Data":"d191459941fb98afd0c0d56b06989c9598375e80b75728687c714eeeacef8d73"} Nov 03 09:02:48.996300 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.996028 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:02:48.997679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.997656 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" event={"ID":"24d00911-860c-454b-b18d-4bed094c4d82","Type":"ContainerStarted","Data":"cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1"} Nov 03 09:02:48.997806 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.997699 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" event={"ID":"24d00911-860c-454b-b18d-4bed094c4d82","Type":"ContainerStarted","Data":"52d2f7d0a1f30140eb0074f9431210e0541c554e95890fd148a536b8089fe6a5"} Nov 03 09:02:48.997806 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:48.997728 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:02:49.116893 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:49.116841 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" podStartSLOduration=2.116826821 podStartE2EDuration="2.116826821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:49.113249365 +0000 UTC m=+1402.038191595" watchObservedRunningTime="2026-11-03 09:02:49.116826821 +0000 UTC m=+1402.041769038" Nov 03 09:02:49.234383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:49.234324 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" podStartSLOduration=2.234308813 podStartE2EDuration="2.234308813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:49.231349744 +0000 UTC m=+1402.156291983" watchObservedRunningTime="2026-11-03 09:02:49.234308813 +0000 UTC m=+1402.159251029" Nov 03 09:02:55.152513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.152446 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97"] Nov 03 09:02:55.157164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.157141 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.213534 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.212607 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d7fd66-openshift-default-dockercfg-ppppf\"" Nov 03 09:02:55.223498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.223000 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-credential-socket\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.223498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.223127 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c2f4a9-66df-4c28-b236-d654ae8a3569-istiod-ca-cert\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.223498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.223169 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-envoy\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.223498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.223218 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-certs\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.223498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.223256 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-podinfo\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.223498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.223287 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-token\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.223498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.223331 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cczs\" (UniqueName: \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-kube-api-access-2cczs\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.223498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.223358 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-socket\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.223498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.223440 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-data\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.294023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.293995 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97"] Nov 03 09:02:55.324785 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.324722 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-certs\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.324979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.324789 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-podinfo\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.324979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.324825 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-token\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.324979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.324873 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cczs\" (UniqueName: \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-kube-api-access-2cczs\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.324979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.324901 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-socket\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.325207 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.324994 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-data\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.325207 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.325070 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-credential-socket\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.325272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.325217 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c2f4a9-66df-4c28-b236-d654ae8a3569-istiod-ca-cert\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.325272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.325262 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-envoy\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.334505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.326138 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-data\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.334505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.326382 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-certs\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.334505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.326466 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-socket\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.334505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.327080 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-credential-socket\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.334505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.328034 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c2f4a9-66df-4c28-b236-d654ae8a3569-istiod-ca-cert\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.334505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.334211 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-envoy\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.337687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.335069 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-podinfo\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.386152 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.385733 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cczs\" (UniqueName: \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-kube-api-access-2cczs\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.388089 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.388040 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-token\") pod \"e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.472723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.472677 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:55.918500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.918386 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97"] Nov 03 09:02:55.956335 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:02:55.956304 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c2f4a9_66df_4c28_b236_d654ae8a3569.slice/crio-38725252b01bd5972b19488ad629855ef4bb5c5f46eec3dd5c1f5efeba5aa9bc WatchSource:0}: Error finding container 38725252b01bd5972b19488ad629855ef4bb5c5f46eec3dd5c1f5efeba5aa9bc: Status 404 returned error can't find the container with id 38725252b01bd5972b19488ad629855ef4bb5c5f46eec3dd5c1f5efeba5aa9bc Nov 03 09:02:55.959033 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.958997 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:55.959132 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.959079 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:55.959206 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:55.959129 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:02:56.056451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:56.056411 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" event={"ID":"52c2f4a9-66df-4c28-b236-d654ae8a3569","Type":"ContainerStarted","Data":"7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969"} Nov 03 09:02:56.056637 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:56.056513 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" event={"ID":"52c2f4a9-66df-4c28-b236-d654ae8a3569","Type":"ContainerStarted","Data":"38725252b01bd5972b19488ad629855ef4bb5c5f46eec3dd5c1f5efeba5aa9bc"} Nov 03 09:02:56.182360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:56.182253 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" podStartSLOduration=1.182232928 podStartE2EDuration="1.182232928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:56.176047431 +0000 UTC m=+1409.100989649" watchObservedRunningTime="2026-11-03 09:02:56.182232928 +0000 UTC m=+1409.107175146" Nov 03 09:02:56.473268 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:56.473231 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:57.486507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:57.485972 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:58.119503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:58.105121 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:58.119503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:58.114725 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:02:58.902501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:58.901596 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:58.944503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:58.943571 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:58.944503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:58.943947 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 09:02:59.118502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:59.117912 5015 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Nov 03 09:02:59.185560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:59.179701 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:59.224642 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:59.217310 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podUID="21ad0cfe-e3c6-4770-b0b3-7c512e7f1830" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:59.224642 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:59.218158 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 09:02:59.906503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:59.904576 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:59.906503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:59.905361 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:02:59.954509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:59.951571 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:59.954509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:59.951951 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 09:02:59.954509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:59.952263 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 09:02:59.954509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:02:59.952322 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 09:03:00.004826 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.004784 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Nov 03 09:03:00.186851 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.186757 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:03:00.234793 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.234760 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:03:00.306126 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.294297 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 03 09:03:00.372007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.306651 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:00.372903 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.372878 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 09:03:00.373097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.373083 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 09:03:00.374072 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.307981 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podUID="21ad0cfe-e3c6-4770-b0b3-7c512e7f1830" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 09:03:00.402825 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.401960 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 09:03:00.544503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.537816 5015 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Nov 03 09:03:00.930557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.927039 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:00.964184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.964136 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:00.986837 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:00.986796 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 09:03:01.034127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:01.018595 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:01.034127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:01.019033 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 09:03:01.411739 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:01.411682 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podUID="21ad0cfe-e3c6-4770-b0b3-7c512e7f1830" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:01.439516 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:01.416729 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 09:03:01.556947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:01.556912 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:03:01.563848 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:01.563808 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 09:03:01.564735 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:01.564711 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 09:03:01.564955 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:01.564943 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 09:03:01.565096 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:01.565086 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 09:03:01.565226 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:01.565216 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 09:03:01.825594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:01.825565 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" Nov 03 09:03:02.185725 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.185640 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84"] Nov 03 09:03:02.193201 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.193172 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.261892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.261837 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-18f9da-openshift-default-dockercfg-zmkbm\"" Nov 03 09:03:02.288829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.288794 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84"] Nov 03 09:03:02.348343 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.348295 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.348764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.348738 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.383598 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.383555 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c714c4-b970-4221-ab53-f6720684ba47-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.383863 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.383840 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.384023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.384006 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.384173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.384154 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c714c4-b970-4221-ab53-f6720684ba47-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.384335 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.384318 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.384451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.384436 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlzgd\" (UniqueName: \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-kube-api-access-rlzgd\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.384631 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.384610 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.496497 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.496547 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c714c4-b970-4221-ab53-f6720684ba47-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.496581 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.496628 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.496686 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c714c4-b970-4221-ab53-f6720684ba47-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.496747 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.496774 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlzgd\" (UniqueName: \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-kube-api-access-rlzgd\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.496827 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.496947 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.497566 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.497825 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.498463 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c714c4-b970-4221-ab53-f6720684ba47-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.499506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.499387 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.501802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.501437 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.567440 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.564541 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.639507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.616254 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.639507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.623127 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c714c4-b970-4221-ab53-f6720684ba47-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.705582 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.705546 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlzgd\" (UniqueName: \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-kube-api-access-rlzgd\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:02.901850 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:02.901752 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:03.440437 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:03:03.440367 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c714c4_b970_4221_ab53_f6720684ba47.slice/crio-2f304ed9f55f197bf1eddcaef48be8e3cccd47477a8380c27459a2ad457504fc WatchSource:0}: Error finding container 2f304ed9f55f197bf1eddcaef48be8e3cccd47477a8380c27459a2ad457504fc: Status 404 returned error can't find the container with id 2f304ed9f55f197bf1eddcaef48be8e3cccd47477a8380c27459a2ad457504fc Nov 03 09:03:03.447517 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:03.441330 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84"] Nov 03 09:03:03.447517 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:03.443816 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:03:03.447517 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:03.443899 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:03:03.447517 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:03.443943 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:03:03.942000 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:03.941919 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" event={"ID":"e9c714c4-b970-4221-ab53-f6720684ba47","Type":"ContainerStarted","Data":"a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966"} Nov 03 09:03:03.943183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:03.943154 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" event={"ID":"e9c714c4-b970-4221-ab53-f6720684ba47","Type":"ContainerStarted","Data":"2f304ed9f55f197bf1eddcaef48be8e3cccd47477a8380c27459a2ad457504fc"} Nov 03 09:03:03.974320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:03.974289 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g"] Nov 03 09:03:03.978596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:03.978573 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:03:04.037944 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.037901 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g"] Nov 03 09:03:04.049423 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.049389 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d7fd66-dockercfg-4sdqv\"" Nov 03 09:03:04.049423 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.049395 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d7fd66\"" Nov 03 09:03:04.082186 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.082144 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa34adb-1c41-4f54-9d38-5f64d37b428c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g\" (UID: \"4aa34adb-1c41-4f54-9d38-5f64d37b428c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:03:04.082397 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.082367 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2z8\" (UniqueName: \"kubernetes.io/projected/4aa34adb-1c41-4f54-9d38-5f64d37b428c-kube-api-access-cb2z8\") pod \"payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g\" (UID: \"4aa34adb-1c41-4f54-9d38-5f64d37b428c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:03:04.085877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.085827 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" podStartSLOduration=2.085813073 podStartE2EDuration="2.085813073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:03:04.066026116 +0000 UTC m=+1416.990968349" watchObservedRunningTime="2026-11-03 09:03:04.085813073 +0000 UTC m=+1417.010755290" Nov 03 09:03:04.183308 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.183265 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2z8\" (UniqueName: \"kubernetes.io/projected/4aa34adb-1c41-4f54-9d38-5f64d37b428c-kube-api-access-cb2z8\") pod \"payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g\" (UID: \"4aa34adb-1c41-4f54-9d38-5f64d37b428c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:03:04.183500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.183390 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa34adb-1c41-4f54-9d38-5f64d37b428c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g\" (UID: \"4aa34adb-1c41-4f54-9d38-5f64d37b428c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:03:04.184207 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.184187 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa34adb-1c41-4f54-9d38-5f64d37b428c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g\" (UID: \"4aa34adb-1c41-4f54-9d38-5f64d37b428c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:03:04.263188 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.263107 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2z8\" (UniqueName: \"kubernetes.io/projected/4aa34adb-1c41-4f54-9d38-5f64d37b428c-kube-api-access-cb2z8\") pod \"payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g\" (UID: \"4aa34adb-1c41-4f54-9d38-5f64d37b428c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:03:04.307883 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.307851 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:03:04.735043 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.735010 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g"] Nov 03 09:03:04.736073 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:03:04.736047 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa34adb_1c41_4f54_9d38_5f64d37b428c.slice/crio-3b198a7a0e4989d5e3b3468235bb2e9eb12ce8e385bbdfdd7c4426c56a7c1755 WatchSource:0}: Error finding container 3b198a7a0e4989d5e3b3468235bb2e9eb12ce8e385bbdfdd7c4426c56a7c1755: Status 404 returned error can't find the container with id 3b198a7a0e4989d5e3b3468235bb2e9eb12ce8e385bbdfdd7c4426c56a7c1755 Nov 03 09:03:04.902891 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.902857 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:04.904694 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.904665 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 09:03:04.904822 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.904717 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 09:03:04.977582 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.977542 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" event={"ID":"4aa34adb-1c41-4f54-9d38-5f64d37b428c","Type":"ContainerStarted","Data":"774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962"} Nov 03 09:03:04.977789 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.977611 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" event={"ID":"4aa34adb-1c41-4f54-9d38-5f64d37b428c","Type":"ContainerStarted","Data":"3b198a7a0e4989d5e3b3468235bb2e9eb12ce8e385bbdfdd7c4426c56a7c1755"} Nov 03 09:03:04.978525 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:04.977903 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:03:05.094966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:05.094855 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" podStartSLOduration=2.094838554 podStartE2EDuration="2.094838554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:03:05.094524292 +0000 UTC m=+1418.019466522" watchObservedRunningTime="2026-11-03 09:03:05.094838554 +0000 UTC m=+1418.019780771" Nov 03 09:03:05.903070 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:05.903000 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 09:03:05.903629 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:05.903106 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 09:03:06.903290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:06.903252 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 09:03:06.903722 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:06.903305 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 09:03:07.903108 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:07.903061 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 09:03:07.903359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:07.903128 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 09:03:08.902889 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:08.902854 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 09:03:08.903089 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:08.902910 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 09:03:09.902680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:09.902645 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 09:03:09.903083 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:09.902702 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 09:03:10.916690 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:10.916658 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:10.917085 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:10.917043 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:10.918261 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:10.918240 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:03:16.034539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:16.033493 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:03:16.903901 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:03:16.903859 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:03:36.308921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.308885 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5"] Nov 03 09:03:36.314604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.314584 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.372498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.372114 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d7fd66-serving-cert\"" Nov 03 09:03:36.372819 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.372793 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5"] Nov 03 09:03:36.381048 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.380996 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8a4bcb-1e03-43d0-9785-8b3c15345594-openshift-service-ca\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.381246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.381136 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-maas-api-tls\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.381246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.381196 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2dsv\" (UniqueName: \"kubernetes.io/projected/2b8a4bcb-1e03-43d0-9785-8b3c15345594-kube-api-access-z2dsv\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.381371 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.381298 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-metrics-certs\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.482886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.482271 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8a4bcb-1e03-43d0-9785-8b3c15345594-openshift-service-ca\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.482886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.482335 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-maas-api-tls\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.482886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.482391 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2dsv\" (UniqueName: \"kubernetes.io/projected/2b8a4bcb-1e03-43d0-9785-8b3c15345594-kube-api-access-z2dsv\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.482886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.482430 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-metrics-certs\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.483685 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:03:36.483657 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-serving-cert: secret "maas-api-e2e-api-a-d7fd66-serving-cert" not found Nov 03 09:03:36.483817 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:03:36.483746 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-maas-api-tls podName:2b8a4bcb-1e03-43d0-9785-8b3c15345594 nodeName:}" failed. No retries permitted until 2026-11-03 09:03:36.983725867 +0000 UTC m=+1449.908668062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-maas-api-tls") pod "maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" (UID: "2b8a4bcb-1e03-43d0-9785-8b3c15345594") : secret "maas-api-e2e-api-a-d7fd66-serving-cert" not found Nov 03 09:03:36.484706 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.484657 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8a4bcb-1e03-43d0-9785-8b3c15345594-openshift-service-ca\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.486672 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.486639 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-metrics-certs\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.587702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.587611 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2dsv\" (UniqueName: \"kubernetes.io/projected/2b8a4bcb-1e03-43d0-9785-8b3c15345594-kube-api-access-z2dsv\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.993302 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.993259 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-maas-api-tls\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:36.995963 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:36.995941 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-maas-api-tls\") pod \"maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:37.228314 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:37.228286 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:37.649528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:37.649467 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5"] Nov 03 09:03:37.669395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:37.669366 5015 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:03:38.551589 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:38.551513 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" event={"ID":"2b8a4bcb-1e03-43d0-9785-8b3c15345594","Type":"ContainerStarted","Data":"38003ed03a813884ad879ca0f10176076019bfc75493e8b6090f388b52c294db"} Nov 03 09:03:39.582528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:39.581620 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" event={"ID":"2b8a4bcb-1e03-43d0-9785-8b3c15345594","Type":"ContainerStarted","Data":"480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654"} Nov 03 09:03:39.583043 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:39.582871 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:39.700567 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:39.700469 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" podStartSLOduration=2.755762352 podStartE2EDuration="3.700449133s" totalImagesPullingTime="944.686781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:03:39.6996381 +0000 UTC m=+1452.624580339" watchObservedRunningTime="2026-11-03 09:03:39.700449133 +0000 UTC m=+1452.625391351" Nov 03 09:03:41.598646 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:41.598613 5015 generic.go:356] "Generic (PLEG): container finished" podID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerID="d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339" exitCode=0 Nov 03 09:03:41.599074 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:41.598682 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" event={"ID":"911aeeac-4f94-4ee1-a6c6-46db527d4f0f","Type":"ContainerDied","Data":"d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339"} Nov 03 09:03:41.599149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:41.599100 5015 scope.go:122] "RemoveContainer" containerID="d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339" Nov 03 09:03:41.600433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:41.600406 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:03:41.600518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:41.600494 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:03:41.600564 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:41.600528 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:03:41.937617 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:41.937543 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:03:42.608180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:42.608137 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" event={"ID":"911aeeac-4f94-4ee1-a6c6-46db527d4f0f","Type":"ContainerStarted","Data":"e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba"} Nov 03 09:03:42.731560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:42.731518 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:03:42.733002 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:42.732973 5015 patch_prober.go:29] interesting pod/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 09:03:42.733155 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:42.733022 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 09:03:43.731776 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:43.731739 5015 patch_prober.go:29] interesting pod/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 09:03:43.732145 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:43.731794 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 09:03:44.731760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:44.731723 5015 patch_prober.go:29] interesting pod/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 09:03:44.731948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:44.731771 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 09:03:45.730914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:45.730880 5015 patch_prober.go:29] interesting pod/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 09:03:45.731089 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:45.730940 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 09:03:46.604443 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:46.604415 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:03:46.731646 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:46.731608 5015 patch_prober.go:29] interesting pod/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 09:03:46.731826 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:46.731668 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 09:03:46.923942 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:03:46.922381 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:03:47.632936 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.632901 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws"] Nov 03 09:03:47.637135 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.637117 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.740344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.740078 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws"] Nov 03 09:03:47.774894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.774849 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6cff466-5838-4ed8-9124-09bdc768a0ca-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.775089 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.774921 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.775089 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.774970 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l5s7\" (UniqueName: \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-kube-api-access-5l5s7\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.775089 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.775016 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.775266 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.775147 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.775266 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.775172 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.775266 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.775210 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.775266 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.775260 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.775489 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.775308 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.821159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.821127 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:03:47.821646 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.821606 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:03:47.822373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.822355 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:03:47.876591 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.876556 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.876790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.876601 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.876790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.876639 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.876790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.876686 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.876790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.876729 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.877022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.876861 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6cff466-5838-4ed8-9124-09bdc768a0ca-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.877022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.876905 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.877022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.876946 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l5s7\" (UniqueName: \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-kube-api-access-5l5s7\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.877022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.876988 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.877595 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.877572 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.877908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.877880 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.877986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.877881 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6cff466-5838-4ed8-9124-09bdc768a0ca-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.877986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.877946 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.878118 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.878092 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.879264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.879242 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.879615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.879594 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.937138 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.937074 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.937287 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.937262 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l5s7\" (UniqueName: \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-kube-api-access-5l5s7\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:47.952454 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:47.952229 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:48.394529 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:03:48.394491 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6cff466_5838_4ed8_9124_09bdc768a0ca.slice/crio-bdbad6126a74ad8ed6805ad9c7cc54d76debe65dc081068f0d22074e6e58bc16 WatchSource:0}: Error finding container bdbad6126a74ad8ed6805ad9c7cc54d76debe65dc081068f0d22074e6e58bc16: Status 404 returned error can't find the container with id bdbad6126a74ad8ed6805ad9c7cc54d76debe65dc081068f0d22074e6e58bc16 Nov 03 09:03:48.395415 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:48.395394 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws"] Nov 03 09:03:48.396985 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:48.396956 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:03:48.397050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:48.397016 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:03:48.397085 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:48.397046 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:03:48.713558 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:48.713373 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" event={"ID":"c6cff466-5838-4ed8-9124-09bdc768a0ca","Type":"ContainerStarted","Data":"ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f"} Nov 03 09:03:48.713558 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:48.713450 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" event={"ID":"c6cff466-5838-4ed8-9124-09bdc768a0ca","Type":"ContainerStarted","Data":"bdbad6126a74ad8ed6805ad9c7cc54d76debe65dc081068f0d22074e6e58bc16"} Nov 03 09:03:48.838050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:48.837997 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" podStartSLOduration=1.837982812 podStartE2EDuration="1.837982812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:03:48.83155774 +0000 UTC m=+1461.756499974" watchObservedRunningTime="2026-11-03 09:03:48.837982812 +0000 UTC m=+1461.762925077" Nov 03 09:03:48.955894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:48.955853 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:49.962800 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:49.962760 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 09:03:49.963251 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:49.962829 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 09:03:50.964523 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:50.963898 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 09:03:50.964523 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:50.963960 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 09:03:51.956918 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:51.956880 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 09:03:51.957222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:51.957195 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 09:03:52.960911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:52.960866 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 09:03:52.969499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:52.965543 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 09:03:53.955247 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:53.955206 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 09:03:53.955639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:53.955591 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 09:03:54.964502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:54.963578 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 09:03:54.964502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:54.963646 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 09:03:56.996618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:56.996572 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:03:56.997123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:56.996639 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:57.668714 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:57.668678 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:57.669150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:57.669120 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:57.670367 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:57.670341 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:03:59.146631 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:59.146589 5015 patch_prober.go:29] interesting pod/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:03:59.147051 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:03:59.146658 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:04:00.893457 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:00.893416 5015 generic.go:356] "Generic (PLEG): container finished" podID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerID="7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969" exitCode=0 Nov 03 09:04:00.893911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:00.893529 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" event={"ID":"52c2f4a9-66df-4c28-b236-d654ae8a3569","Type":"ContainerDied","Data":"7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969"} Nov 03 09:04:00.894104 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:00.894086 5015 scope.go:122] "RemoveContainer" containerID="7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969" Nov 03 09:04:00.895785 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:00.895751 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:04:00.895929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:00.895840 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:04:00.895929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:00.895886 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:04:01.911360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:01.911318 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" event={"ID":"52c2f4a9-66df-4c28-b236-d654ae8a3569","Type":"ContainerStarted","Data":"4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc"} Nov 03 09:04:02.473353 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:02.473323 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:04:02.473585 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:02.473449 5015 patch_prober.go:29] interesting pod/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 09:04:02.473585 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:02.473520 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 09:04:03.473850 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:03.473808 5015 patch_prober.go:29] interesting pod/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 09:04:03.474350 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:03.473868 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 09:04:04.473123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:04.473088 5015 patch_prober.go:29] interesting pod/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 09:04:04.473325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:04.473149 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 09:04:05.473748 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:05.473665 5015 patch_prober.go:29] interesting pod/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 09:04:05.473748 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:05.473721 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 09:04:06.474066 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:06.474031 5015 patch_prober.go:29] interesting pod/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 09:04:06.474561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:06.474093 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 09:04:07.478066 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:07.478035 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:04:07.478525 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:07.478504 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:04:07.479239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:07.479222 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:04:16.564444 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.564407 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb"] Nov 03 09:04:16.568797 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.568774 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:04:16.615002 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.614970 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c50c64-e1d9-4f70-bf80-12713de54c41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb\" (UID: \"97c50c64-e1d9-4f70-bf80-12713de54c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:04:16.615165 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.615106 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlkbl\" (UniqueName: \"kubernetes.io/projected/97c50c64-e1d9-4f70-bf80-12713de54c41-kube-api-access-hlkbl\") pod \"payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb\" (UID: \"97c50c64-e1d9-4f70-bf80-12713de54c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:04:16.627267 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.627235 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb"] Nov 03 09:04:16.688890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.688853 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-204dfc\"" Nov 03 09:04:16.701803 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.689358 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-204dfc-dockercfg-pvt7s\"" Nov 03 09:04:16.759695 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.746892 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c50c64-e1d9-4f70-bf80-12713de54c41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb\" (UID: \"97c50c64-e1d9-4f70-bf80-12713de54c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:04:16.759695 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.747047 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkbl\" (UniqueName: \"kubernetes.io/projected/97c50c64-e1d9-4f70-bf80-12713de54c41-kube-api-access-hlkbl\") pod \"payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb\" (UID: \"97c50c64-e1d9-4f70-bf80-12713de54c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:04:16.759695 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.748329 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c50c64-e1d9-4f70-bf80-12713de54c41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb\" (UID: \"97c50c64-e1d9-4f70-bf80-12713de54c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:04:16.806708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.806669 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld"] Nov 03 09:04:16.811638 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.811611 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:04:16.817145 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.817079 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld"] Nov 03 09:04:16.821597 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.821568 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlkbl\" (UniqueName: \"kubernetes.io/projected/97c50c64-e1d9-4f70-bf80-12713de54c41-kube-api-access-hlkbl\") pod \"payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb\" (UID: \"97c50c64-e1d9-4f70-bf80-12713de54c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:04:16.894079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.894047 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:04:16.904142 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:04:16.904113 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:04:16.957585 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.954342 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9c75c1-ec86-4637-be65-48efdda20ba7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld\" (UID: \"4f9c75c1-ec86-4637-be65-48efdda20ba7\") " pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:04:16.957585 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:16.954393 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql9st\" (UniqueName: \"kubernetes.io/projected/4f9c75c1-ec86-4637-be65-48efdda20ba7-kube-api-access-ql9st\") pod \"payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld\" (UID: \"4f9c75c1-ec86-4637-be65-48efdda20ba7\") " pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:04:17.056717 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.056680 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9c75c1-ec86-4637-be65-48efdda20ba7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld\" (UID: \"4f9c75c1-ec86-4637-be65-48efdda20ba7\") " pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:04:17.056879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.056728 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql9st\" (UniqueName: \"kubernetes.io/projected/4f9c75c1-ec86-4637-be65-48efdda20ba7-kube-api-access-ql9st\") pod \"payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld\" (UID: \"4f9c75c1-ec86-4637-be65-48efdda20ba7\") " pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:04:17.057641 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.057615 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9c75c1-ec86-4637-be65-48efdda20ba7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld\" (UID: \"4f9c75c1-ec86-4637-be65-48efdda20ba7\") " pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:04:17.117027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.116956 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql9st\" (UniqueName: \"kubernetes.io/projected/4f9c75c1-ec86-4637-be65-48efdda20ba7-kube-api-access-ql9st\") pod \"payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld\" (UID: \"4f9c75c1-ec86-4637-be65-48efdda20ba7\") " pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:04:17.162528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.162459 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:04:17.292157 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.292051 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb"] Nov 03 09:04:17.547797 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:04:17.547770 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9c75c1_ec86_4637_be65_48efdda20ba7.slice/crio-20a1a9bdde59eec58613696a77a980b4302d5417080f50832b1aa7f448332361 WatchSource:0}: Error finding container 20a1a9bdde59eec58613696a77a980b4302d5417080f50832b1aa7f448332361: Status 404 returned error can't find the container with id 20a1a9bdde59eec58613696a77a980b4302d5417080f50832b1aa7f448332361 Nov 03 09:04:17.548184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.548165 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld"] Nov 03 09:04:17.876795 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.876708 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s"] Nov 03 09:04:17.881022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.880992 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:17.969578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.969543 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxmsw\" (UniqueName: \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-kube-api-access-cxmsw\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:17.969789 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.969599 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:17.969789 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.969719 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:17.969922 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.969853 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:17.969993 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.969915 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f6d302-4730-41f8-88ca-410efbf69624-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:17.969993 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.969971 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:17.970114 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.970041 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:17.970114 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.970099 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:17.970219 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.970140 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f6d302-4730-41f8-88ca-410efbf69624-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:17.984285 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:17.984084 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s"] Nov 03 09:04:18.071376 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.071335 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxmsw\" (UniqueName: \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-kube-api-access-cxmsw\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.071651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.071392 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.071651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.071458 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.071651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.071584 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.071651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.071618 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f6d302-4730-41f8-88ca-410efbf69624-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.071902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.071666 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.071902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.071739 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.071902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.071799 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.071902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.071835 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f6d302-4730-41f8-88ca-410efbf69624-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.071902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.071870 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.072224 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.072201 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.072462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.072437 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f6d302-4730-41f8-88ca-410efbf69624-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.072600 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.072558 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.072708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.072687 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.075083 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.075055 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.075202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.075059 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f6d302-4730-41f8-88ca-410efbf69624-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.123859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.123054 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" event={"ID":"4f9c75c1-ec86-4637-be65-48efdda20ba7","Type":"ContainerStarted","Data":"89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21"} Nov 03 09:04:18.123859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.123116 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" event={"ID":"4f9c75c1-ec86-4637-be65-48efdda20ba7","Type":"ContainerStarted","Data":"20a1a9bdde59eec58613696a77a980b4302d5417080f50832b1aa7f448332361"} Nov 03 09:04:18.123859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.123200 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:04:18.125113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.125083 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" event={"ID":"97c50c64-e1d9-4f70-bf80-12713de54c41","Type":"ContainerStarted","Data":"22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2"} Nov 03 09:04:18.125278 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.125135 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" event={"ID":"97c50c64-e1d9-4f70-bf80-12713de54c41","Type":"ContainerStarted","Data":"ad2e62f887b0204de48d1df686aa85e1085b943569b17803beb6719baaee44cb"} Nov 03 09:04:18.125278 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.125207 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:04:18.131161 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.131107 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.131418 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.131394 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxmsw\" (UniqueName: \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-kube-api-access-cxmsw\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.195945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.195915 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:18.246804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.246750 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" podStartSLOduration=2.246732443 podStartE2EDuration="2.246732443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:18.243561366 +0000 UTC m=+1491.168503585" watchObservedRunningTime="2026-11-03 09:04:18.246732443 +0000 UTC m=+1491.171675058" Nov 03 09:04:18.364531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.364454 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" podStartSLOduration=2.364440247 podStartE2EDuration="2.364440247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:18.359809669 +0000 UTC m=+1491.284751889" watchObservedRunningTime="2026-11-03 09:04:18.364440247 +0000 UTC m=+1491.289382469" Nov 03 09:04:18.642575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.642535 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s"] Nov 03 09:04:18.650539 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:04:18.649995 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f6d302_4730_41f8_88ca_410efbf69624.slice/crio-fa6582224c39848a7d857abfee880bd9de120591f3f7cf63a01940cb6801c7d1 WatchSource:0}: Error finding container fa6582224c39848a7d857abfee880bd9de120591f3f7cf63a01940cb6801c7d1: Status 404 returned error can't find the container with id fa6582224c39848a7d857abfee880bd9de120591f3f7cf63a01940cb6801c7d1 Nov 03 09:04:18.653828 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.653770 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:04:18.653968 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.653866 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:04:18.653968 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:18.653908 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:04:19.148999 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:19.148965 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" event={"ID":"27f6d302-4730-41f8-88ca-410efbf69624","Type":"ContainerStarted","Data":"d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582"} Nov 03 09:04:19.149685 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:19.149659 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" event={"ID":"27f6d302-4730-41f8-88ca-410efbf69624","Type":"ContainerStarted","Data":"fa6582224c39848a7d857abfee880bd9de120591f3f7cf63a01940cb6801c7d1"} Nov 03 09:04:19.198447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:19.198412 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 09:04:19.198760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:19.198729 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" podUID="27f6d302-4730-41f8-88ca-410efbf69624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 09:04:19.208085 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:19.208062 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:19.278905 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:19.276901 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" podStartSLOduration=2.276879165 podStartE2EDuration="2.276879165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:19.267712958 +0000 UTC m=+1492.192655222" watchObservedRunningTime="2026-11-03 09:04:19.276879165 +0000 UTC m=+1492.201821382" Nov 03 09:04:20.197307 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:20.197186 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 09:04:20.197959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:20.197921 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" podUID="27f6d302-4730-41f8-88ca-410efbf69624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 09:04:21.196350 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:21.196312 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 09:04:21.196573 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:21.196375 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" podUID="27f6d302-4730-41f8-88ca-410efbf69624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 09:04:22.196569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:22.196532 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 09:04:22.197087 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:22.196590 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" podUID="27f6d302-4730-41f8-88ca-410efbf69624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 09:04:23.196803 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:23.196766 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 09:04:23.197317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:23.197229 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" podUID="27f6d302-4730-41f8-88ca-410efbf69624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 09:04:24.196811 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:24.196771 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 09:04:24.197204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:24.196845 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" podUID="27f6d302-4730-41f8-88ca-410efbf69624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 09:04:25.196288 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:25.196254 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 09:04:25.196535 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:25.196317 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" podUID="27f6d302-4730-41f8-88ca-410efbf69624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 09:04:26.223416 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:26.223351 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:26.224470 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:26.224404 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:26.225713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:26.225691 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:04:27.461004 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:27.460975 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:04:27.468770 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:27.468743 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:04:29.151671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:29.151643 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:04:29.153110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:29.153087 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:04:34.519636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.519599 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh"] Nov 03 09:04:34.526656 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.526628 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:04:34.577730 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.577698 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh"] Nov 03 09:04:34.596777 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.596692 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df586e68-525d-4a8a-9926-fe81845c9be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh\" (UID: \"df586e68-525d-4a8a-9926-fe81845c9be9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:04:34.596973 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.596938 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn9gz\" (UniqueName: \"kubernetes.io/projected/df586e68-525d-4a8a-9926-fe81845c9be9-kube-api-access-jn9gz\") pod \"payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh\" (UID: \"df586e68-525d-4a8a-9926-fe81845c9be9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:04:34.661641 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.625676 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b8fd34\"" Nov 03 09:04:34.662153 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.629846 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b8fd34-dockercfg-lflzg\"" Nov 03 09:04:34.698869 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.698833 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df586e68-525d-4a8a-9926-fe81845c9be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh\" (UID: \"df586e68-525d-4a8a-9926-fe81845c9be9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:04:34.699936 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.699911 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9gz\" (UniqueName: \"kubernetes.io/projected/df586e68-525d-4a8a-9926-fe81845c9be9-kube-api-access-jn9gz\") pod \"payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh\" (UID: \"df586e68-525d-4a8a-9926-fe81845c9be9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:04:34.700989 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.700968 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df586e68-525d-4a8a-9926-fe81845c9be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh\" (UID: \"df586e68-525d-4a8a-9926-fe81845c9be9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:04:34.781683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.781566 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9gz\" (UniqueName: \"kubernetes.io/projected/df586e68-525d-4a8a-9926-fe81845c9be9-kube-api-access-jn9gz\") pod \"payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh\" (UID: \"df586e68-525d-4a8a-9926-fe81845c9be9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:04:34.843693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:34.843665 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:04:35.232633 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:35.232594 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh"] Nov 03 09:04:35.318848 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:35.318810 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" event={"ID":"df586e68-525d-4a8a-9926-fe81845c9be9","Type":"ContainerStarted","Data":"18c544a7c321dd8b2c81334c9254a4e74f057991cecbb61cd86ed8f7d05621ba"} Nov 03 09:04:36.328651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:36.328611 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" event={"ID":"df586e68-525d-4a8a-9926-fe81845c9be9","Type":"ContainerStarted","Data":"098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9"} Nov 03 09:04:36.329147 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:36.328808 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:04:36.445768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:36.445633 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" podStartSLOduration=2.445611575 podStartE2EDuration="2.445611575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:36.444923713 +0000 UTC m=+1509.369865933" watchObservedRunningTime="2026-11-03 09:04:36.445611575 +0000 UTC m=+1509.370553792" Nov 03 09:04:37.616319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:37.616277 5015 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" podUID="6652f123a319d675474cd7afd732c542" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:04:46.902531 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:04:46.902456 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:04:47.339887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.339860 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:04:47.714680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.714595 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx"] Nov 03 09:04:47.719025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.719008 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.785841 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.785806 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhtt\" (UniqueName: \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-kube-api-access-phhtt\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.786037 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.785956 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5936599-d8f2-4705-9152-81bdabf3319e-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.786037 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.786017 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.786151 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.786050 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.786151 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.786131 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.786225 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.786157 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5936599-d8f2-4705-9152-81bdabf3319e-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.786328 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.786303 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.786420 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.786385 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.786489 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.786430 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.828708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.828665 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx"] Nov 03 09:04:47.887792 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.887759 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.887960 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.887818 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.887960 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.887852 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.888076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.888011 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhtt\" (UniqueName: \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-kube-api-access-phhtt\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.888173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.888153 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5936599-d8f2-4705-9152-81bdabf3319e-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.888540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.888222 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.888540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.888238 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.888540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.888227 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.888540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.888285 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.888540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.888351 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.888540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.888379 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5936599-d8f2-4705-9152-81bdabf3319e-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.888940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.888700 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.888940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.888792 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.889027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.888977 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5936599-d8f2-4705-9152-81bdabf3319e-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.890300 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.890278 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.891224 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.891205 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5936599-d8f2-4705-9152-81bdabf3319e-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.950339 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.950299 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:47.950746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:47.950352 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhtt\" (UniqueName: \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-kube-api-access-phhtt\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:48.033296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:48.033252 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:48.482351 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:04:48.482318 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5936599_d8f2_4705_9152_81bdabf3319e.slice/crio-15a1d1ad541bf89af24c29bc4e7b15fcd95f0ae179d6867cc16465ef0da53272 WatchSource:0}: Error finding container 15a1d1ad541bf89af24c29bc4e7b15fcd95f0ae179d6867cc16465ef0da53272: Status 404 returned error can't find the container with id 15a1d1ad541bf89af24c29bc4e7b15fcd95f0ae179d6867cc16465ef0da53272 Nov 03 09:04:48.482520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:48.482424 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx"] Nov 03 09:04:48.485578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:48.485545 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:04:48.485692 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:48.485626 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:04:48.485692 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:48.485673 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:04:49.478736 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:49.478692 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" event={"ID":"c5936599-d8f2-4705-9152-81bdabf3319e","Type":"ContainerStarted","Data":"313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6"} Nov 03 09:04:49.479316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:49.478773 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" event={"ID":"c5936599-d8f2-4705-9152-81bdabf3319e","Type":"ContainerStarted","Data":"15a1d1ad541bf89af24c29bc4e7b15fcd95f0ae179d6867cc16465ef0da53272"} Nov 03 09:04:49.665461 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:49.665387 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" podStartSLOduration=2.665370082 podStartE2EDuration="2.665370082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:49.649959588 +0000 UTC m=+1522.574901805" watchObservedRunningTime="2026-11-03 09:04:49.665370082 +0000 UTC m=+1522.590312304" Nov 03 09:04:50.034358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:50.034324 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:50.035941 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:50.035915 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 09:04:50.036182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:50.035976 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" podUID="c5936599-d8f2-4705-9152-81bdabf3319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 09:04:51.033846 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:51.033812 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 09:04:51.034238 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:51.033889 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" podUID="c5936599-d8f2-4705-9152-81bdabf3319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 09:04:52.034353 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:52.034318 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 09:04:52.034888 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:52.034391 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" podUID="c5936599-d8f2-4705-9152-81bdabf3319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 09:04:53.034181 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:53.034149 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 09:04:53.034360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:53.034201 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" podUID="c5936599-d8f2-4705-9152-81bdabf3319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 09:04:54.033757 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:54.033720 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 09:04:54.034173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:54.033777 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" podUID="c5936599-d8f2-4705-9152-81bdabf3319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 09:04:54.910644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:54.910605 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg"] Nov 03 09:04:54.916296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:54.916270 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:54.922713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:54.922683 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr4z5\" (UniqueName: \"kubernetes.io/projected/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-kube-api-access-gr4z5\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:54.922843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:54.922757 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-openshift-service-ca\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:54.922943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:54.922917 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-maas-api-tls\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:54.923017 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:54.922976 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-metrics-certs\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:54.971442 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:54.971406 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-204dfc-serving-cert\"" Nov 03 09:04:55.044833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.026000 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg"] Nov 03 09:04:55.044833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.027394 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4z5\" (UniqueName: \"kubernetes.io/projected/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-kube-api-access-gr4z5\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:55.044833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.027448 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-openshift-service-ca\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:55.044833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.027530 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-maas-api-tls\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:55.044833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.027561 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-metrics-certs\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:55.044833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.028310 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-openshift-service-ca\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:55.044833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.030460 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-metrics-certs\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:55.044833 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:04:55.030574 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-serving-cert: secret "maas-api-e2e-api-a-204dfc-serving-cert" not found Nov 03 09:04:55.044833 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:04:55.030643 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-maas-api-tls podName:90342bf6-bae8-4bc6-8fe4-c538d0ac63f6 nodeName:}" failed. No retries permitted until 2026-11-03 09:04:55.530625541 +0000 UTC m=+1528.455567736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-maas-api-tls") pod "maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" (UID: "90342bf6-bae8-4bc6-8fe4-c538d0ac63f6") : secret "maas-api-e2e-api-a-204dfc-serving-cert" not found Nov 03 09:04:55.044833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.043202 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 09:04:55.044833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.043248 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" podUID="c5936599-d8f2-4705-9152-81bdabf3319e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 09:04:55.089418 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.089392 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4z5\" (UniqueName: \"kubernetes.io/projected/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-kube-api-access-gr4z5\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:55.535509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.535456 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-maas-api-tls\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:55.538886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.538852 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-maas-api-tls\") pod \"maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:55.832283 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:55.832177 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:04:56.039168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:56.039143 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:56.039602 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:56.039586 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:56.040425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:56.040407 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:04:56.326790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:56.326750 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg"] Nov 03 09:04:56.544635 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:56.544592 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" event={"ID":"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6","Type":"ContainerStarted","Data":"799c281a44c11da1f2bbdd8b925ed75c60e67f89a180dcdc65617a31ad9aa7d3"} Nov 03 09:04:57.572536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:57.572389 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" event={"ID":"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6","Type":"ContainerStarted","Data":"5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457"} Nov 03 09:04:57.573011 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:04:57.572546 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:05:03.594254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:03.594225 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:05:03.713485 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:03.713425 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" podStartSLOduration=8.836754157 podStartE2EDuration="9.713412118s" totalImagesPullingTime="876.657961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:57.705050465 +0000 UTC m=+1530.629992699" watchObservedRunningTime="2026-11-03 09:05:03.713412118 +0000 UTC m=+1536.638354371" Nov 03 09:05:15.494514 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.494054 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd"] Nov 03 09:05:15.516750 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.516720 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:15.590546 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.590509 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd"] Nov 03 09:05:15.594759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.594728 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b8fd34-serving-cert\"" Nov 03 09:05:15.596594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.596397 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac04ca7-4fa9-4350-878f-5716bbde17ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:15.596594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.596440 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9lgq\" (UniqueName: \"kubernetes.io/projected/cac04ca7-4fa9-4350-878f-5716bbde17ba-kube-api-access-x9lgq\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:15.596780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.596697 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-metrics-certs\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:15.596780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.596766 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-maas-api-tls\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:15.840559 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.836736 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-metrics-certs\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:15.932580 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.932542 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-maas-api-tls\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:15.934842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.908095 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-metrics-certs\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:15.934842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.933413 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac04ca7-4fa9-4350-878f-5716bbde17ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:15.934842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.933468 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9lgq\" (UniqueName: \"kubernetes.io/projected/cac04ca7-4fa9-4350-878f-5716bbde17ba-kube-api-access-x9lgq\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:15.934842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:15.934792 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac04ca7-4fa9-4350-878f-5716bbde17ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:16.055703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:16.055616 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9lgq\" (UniqueName: \"kubernetes.io/projected/cac04ca7-4fa9-4350-878f-5716bbde17ba-kube-api-access-x9lgq\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:16.219627 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:16.201133 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-maas-api-tls\") pod \"maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:16.508503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:16.502186 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:16.914724 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:05:16.914639 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:05:17.054092 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:05:17.054061 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac04ca7_4fa9_4350_878f_5716bbde17ba.slice/crio-a39329b8f963c314b73830f5bf70f18073950d2793117a0d6a1fa64f0be50646 WatchSource:0}: Error finding container a39329b8f963c314b73830f5bf70f18073950d2793117a0d6a1fa64f0be50646: Status 404 returned error can't find the container with id a39329b8f963c314b73830f5bf70f18073950d2793117a0d6a1fa64f0be50646 Nov 03 09:05:17.054925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:17.054905 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd"] Nov 03 09:05:18.035704 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:18.035668 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" event={"ID":"cac04ca7-4fa9-4350-878f-5716bbde17ba","Type":"ContainerStarted","Data":"c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9"} Nov 03 09:05:18.036173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:18.035729 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" event={"ID":"cac04ca7-4fa9-4350-878f-5716bbde17ba","Type":"ContainerStarted","Data":"a39329b8f963c314b73830f5bf70f18073950d2793117a0d6a1fa64f0be50646"} Nov 03 09:05:18.036173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:18.035757 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:18.155045 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:18.154922 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" podStartSLOduration=2.375421146 podStartE2EDuration="3.154904709s" totalImagesPullingTime="779.483563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:05:18.152534713 +0000 UTC m=+1551.077476949" watchObservedRunningTime="2026-11-03 09:05:18.154904709 +0000 UTC m=+1551.079846927" Nov 03 09:05:24.046159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:24.046117 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" podUID="cac04ca7-4fa9-4350-878f-5716bbde17ba" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.125:8443/health\": dial tcp 10.132.0.125:8443: connect: connection refused" Nov 03 09:05:28.119411 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:28.119377 5015 generic.go:356] "Generic (PLEG): container finished" podID="cac04ca7-4fa9-4350-878f-5716bbde17ba" containerID="c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9" exitCode=1 Nov 03 09:05:28.119832 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:28.119454 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" event={"ID":"cac04ca7-4fa9-4350-878f-5716bbde17ba","Type":"ContainerDied","Data":"c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9"} Nov 03 09:05:28.119938 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:28.119924 5015 scope.go:122] "RemoveContainer" containerID="c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9" Nov 03 09:05:29.128386 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:29.128353 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" event={"ID":"cac04ca7-4fa9-4350-878f-5716bbde17ba","Type":"ContainerStarted","Data":"f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228"} Nov 03 09:05:29.128813 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:29.128550 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:35.142608 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:05:35.142579 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:05:46.904548 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:05:46.904470 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:06:03.058297 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:03.058262 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd"] Nov 03 09:06:03.108123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:03.108091 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8bfa982-lnrcd"] Nov 03 09:06:03.208123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:03.208071 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c79283dd-fcdb-4c11-9796-9deb0f836c91" path="/var/lib/kubelet/pods/c79283dd-fcdb-4c11-9796-9deb0f836c91/volumes" Nov 03 09:06:16.903974 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:06:16.903935 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:06:18.083078 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.083045 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf"] Nov 03 09:06:18.089587 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.089022 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:06:18.148404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.148375 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf"] Nov 03 09:06:18.179372 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.179339 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf\" (UID: \"ba4cfbfa-394c-4ce2-bf0e-327e195c0637\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:06:18.179866 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.179842 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlz98\" (UniqueName: \"kubernetes.io/projected/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-kube-api-access-nlz98\") pod \"payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf\" (UID: \"ba4cfbfa-394c-4ce2-bf0e-327e195c0637\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:06:18.218753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.218713 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-eb9cd2-dockercfg-22nkr\"" Nov 03 09:06:18.229753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.229722 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-eb9cd2\"" Nov 03 09:06:18.278695 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.278650 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8"] Nov 03 09:06:18.285346 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.285240 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf\" (UID: \"ba4cfbfa-394c-4ce2-bf0e-327e195c0637\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:06:18.285995 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.285967 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf\" (UID: \"ba4cfbfa-394c-4ce2-bf0e-327e195c0637\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:06:18.286493 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.286452 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlz98\" (UniqueName: \"kubernetes.io/projected/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-kube-api-access-nlz98\") pod \"payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf\" (UID: \"ba4cfbfa-394c-4ce2-bf0e-327e195c0637\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:06:18.292909 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.292888 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:06:18.308729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.308701 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8"] Nov 03 09:06:18.388975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.388877 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjct\" (UniqueName: \"kubernetes.io/projected/06fca9f0-42e9-4164-9b25-8226f5644f12-kube-api-access-scjct\") pod \"payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8\" (UID: \"06fca9f0-42e9-4164-9b25-8226f5644f12\") " pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:06:18.389291 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.389265 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fca9f0-42e9-4164-9b25-8226f5644f12-plugins-config-volume\") pod \"payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8\" (UID: \"06fca9f0-42e9-4164-9b25-8226f5644f12\") " pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:06:18.409246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.409210 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlz98\" (UniqueName: \"kubernetes.io/projected/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-kube-api-access-nlz98\") pod \"payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf\" (UID: \"ba4cfbfa-394c-4ce2-bf0e-327e195c0637\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:06:18.419390 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.419357 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:06:18.491510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.491184 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjct\" (UniqueName: \"kubernetes.io/projected/06fca9f0-42e9-4164-9b25-8226f5644f12-kube-api-access-scjct\") pod \"payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8\" (UID: \"06fca9f0-42e9-4164-9b25-8226f5644f12\") " pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:06:18.491510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.491283 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fca9f0-42e9-4164-9b25-8226f5644f12-plugins-config-volume\") pod \"payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8\" (UID: \"06fca9f0-42e9-4164-9b25-8226f5644f12\") " pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:06:18.491981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.491961 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fca9f0-42e9-4164-9b25-8226f5644f12-plugins-config-volume\") pod \"payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8\" (UID: \"06fca9f0-42e9-4164-9b25-8226f5644f12\") " pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:06:18.551135 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.551097 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjct\" (UniqueName: \"kubernetes.io/projected/06fca9f0-42e9-4164-9b25-8226f5644f12-kube-api-access-scjct\") pod \"payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8\" (UID: \"06fca9f0-42e9-4164-9b25-8226f5644f12\") " pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:06:18.642503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.642399 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:06:18.836032 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:06:18.836004 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4cfbfa_394c_4ce2_bf0e_327e195c0637.slice/crio-61ae2111be273a47bbd41b026370c605fa7ca3181c97a3a0a05be72469d0a345 WatchSource:0}: Error finding container 61ae2111be273a47bbd41b026370c605fa7ca3181c97a3a0a05be72469d0a345: Status 404 returned error can't find the container with id 61ae2111be273a47bbd41b026370c605fa7ca3181c97a3a0a05be72469d0a345 Nov 03 09:06:18.837197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:18.837174 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf"] Nov 03 09:06:19.033122 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:19.033096 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8"] Nov 03 09:06:19.033235 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:06:19.033210 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fca9f0_42e9_4164_9b25_8226f5644f12.slice/crio-a99797349e8c25cd59deb3783ef5ba53ad2083056138781b624b6c2f67dad40f WatchSource:0}: Error finding container a99797349e8c25cd59deb3783ef5ba53ad2083056138781b624b6c2f67dad40f: Status 404 returned error can't find the container with id a99797349e8c25cd59deb3783ef5ba53ad2083056138781b624b6c2f67dad40f Nov 03 09:06:19.627661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:19.627623 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" event={"ID":"ba4cfbfa-394c-4ce2-bf0e-327e195c0637","Type":"ContainerStarted","Data":"05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7"} Nov 03 09:06:19.628118 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:19.627684 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" event={"ID":"ba4cfbfa-394c-4ce2-bf0e-327e195c0637","Type":"ContainerStarted","Data":"61ae2111be273a47bbd41b026370c605fa7ca3181c97a3a0a05be72469d0a345"} Nov 03 09:06:19.628118 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:19.627712 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:06:19.629328 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:19.629301 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" event={"ID":"06fca9f0-42e9-4164-9b25-8226f5644f12","Type":"ContainerStarted","Data":"334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48"} Nov 03 09:06:19.629420 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:19.629352 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" event={"ID":"06fca9f0-42e9-4164-9b25-8226f5644f12","Type":"ContainerStarted","Data":"a99797349e8c25cd59deb3783ef5ba53ad2083056138781b624b6c2f67dad40f"} Nov 03 09:06:19.629518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:19.629421 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:06:19.748596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:19.748545 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" podStartSLOduration=1.748531246 podStartE2EDuration="1.748531246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:06:19.744180851 +0000 UTC m=+1612.669123068" watchObservedRunningTime="2026-11-03 09:06:19.748531246 +0000 UTC m=+1612.673473463" Nov 03 09:06:19.866174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:19.866114 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" podStartSLOduration=1.8660939829999998 podStartE2EDuration="1.866093983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:06:19.864203553 +0000 UTC m=+1612.789145802" watchObservedRunningTime="2026-11-03 09:06:19.866093983 +0000 UTC m=+1612.791036221" Nov 03 09:06:26.286407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:26.286356 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g"] Nov 03 09:06:30.640157 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:30.640124 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:06:30.640617 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:30.640216 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:06:31.287741 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.287693 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" podUID="4aa34adb-1c41-4f54-9d38-5f64d37b428c" containerName="payload-pre-processing" containerID="cri-o://774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962" gracePeriod=25 Nov 03 09:06:31.611290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.611242 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:06:31.717167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.717139 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa34adb-1c41-4f54-9d38-5f64d37b428c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa34adb-1c41-4f54-9d38-5f64d37b428c-plugins-config-volume\") pod \"4aa34adb-1c41-4f54-9d38-5f64d37b428c\" (UID: \"4aa34adb-1c41-4f54-9d38-5f64d37b428c\") " Nov 03 09:06:31.717629 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.717323 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa34adb-1c41-4f54-9d38-5f64d37b428c-kube-api-access-cb2z8\" (UniqueName: \"kubernetes.io/projected/4aa34adb-1c41-4f54-9d38-5f64d37b428c-kube-api-access-cb2z8\") pod \"4aa34adb-1c41-4f54-9d38-5f64d37b428c\" (UID: \"4aa34adb-1c41-4f54-9d38-5f64d37b428c\") " Nov 03 09:06:31.717629 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.717554 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa34adb-1c41-4f54-9d38-5f64d37b428c-plugins-config-volume" pod "4aa34adb-1c41-4f54-9d38-5f64d37b428c" (UID: "4aa34adb-1c41-4f54-9d38-5f64d37b428c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:06:31.717855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.717839 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa34adb-1c41-4f54-9d38-5f64d37b428c-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:31.720083 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.720062 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa34adb-1c41-4f54-9d38-5f64d37b428c-kube-api-access-cb2z8" pod "4aa34adb-1c41-4f54-9d38-5f64d37b428c" (UID: "4aa34adb-1c41-4f54-9d38-5f64d37b428c"). InnerVolumeSpecName "kube-api-access-cb2z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:06:31.774039 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.773999 5015 generic.go:356] "Generic (PLEG): container finished" podID="4aa34adb-1c41-4f54-9d38-5f64d37b428c" containerID="774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962" exitCode=0 Nov 03 09:06:31.774308 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.774289 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" event={"ID":"4aa34adb-1c41-4f54-9d38-5f64d37b428c","Type":"ContainerDied","Data":"774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962"} Nov 03 09:06:31.774401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.774344 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" event={"ID":"4aa34adb-1c41-4f54-9d38-5f64d37b428c","Type":"ContainerDied","Data":"3b198a7a0e4989d5e3b3468235bb2e9eb12ce8e385bbdfdd7c4426c56a7c1755"} Nov 03 09:06:31.774401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.774372 5015 scope.go:122] "RemoveContainer" containerID="774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962" Nov 03 09:06:31.774588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.774560 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g" Nov 03 09:06:31.795104 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.795077 5015 scope.go:122] "RemoveContainer" containerID="774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962" Nov 03 09:06:31.795421 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:06:31.795400 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962\": container with ID starting with 774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962 not found: ID does not exist" containerID="774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962" Nov 03 09:06:31.795468 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.795429 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962"} err="failed to get container status \"774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962\": rpc error: code = NotFound desc = could not find container \"774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962\": container with ID starting with 774a9042b9c8a59ca0798b19f77b5169478d1e661a83882c92c3178cd987e962 not found: ID does not exist" Nov 03 09:06:31.819016 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.818986 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb2z8\" (UniqueName: \"kubernetes.io/projected/4aa34adb-1c41-4f54-9d38-5f64d37b428c-kube-api-access-cb2z8\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:31.956163 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.956073 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g"] Nov 03 09:06:31.964531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:31.964064 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fd66-76bbf48995-z7x6g"] Nov 03 09:06:33.207043 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:33.206988 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa34adb-1c41-4f54-9d38-5f64d37b428c" path="/var/lib/kubelet/pods/4aa34adb-1c41-4f54-9d38-5f64d37b428c/volumes" Nov 03 09:06:33.283035 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:33.282995 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf"] Nov 03 09:06:33.394682 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:33.394542 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf"] Nov 03 09:06:37.920867 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:37.915364 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g"] Nov 03 09:06:37.932645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:37.932610 5015 cpu_manager.go:414] "containerMap: removing container" podUID="4aa34adb-1c41-4f54-9d38-5f64d37b428c" containerName="payload-pre-processing" Nov 03 09:06:37.932885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:37.932862 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa34adb-1c41-4f54-9d38-5f64d37b428c" containerName="payload-pre-processing" Nov 03 09:06:37.950837 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:37.948228 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa34adb-1c41-4f54-9d38-5f64d37b428c" containerName="payload-pre-processing" Nov 03 09:06:37.954318 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:37.954292 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:06:37.966569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:37.965675 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g"] Nov 03 09:06:38.062384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.062352 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-840559-dockercfg-kg87z\"" Nov 03 09:06:38.062576 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.062539 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-840559\"" Nov 03 09:06:38.067307 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.067285 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pkpk\" (UniqueName: \"kubernetes.io/projected/be58420c-fc67-4845-9088-65718c449087-kube-api-access-4pkpk\") pod \"payload-processing-e2e-api-a-840559-65698dd65c-8d26g\" (UID: \"be58420c-fc67-4845-9088-65718c449087\") " pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:06:38.067640 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.067620 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be58420c-fc67-4845-9088-65718c449087-plugins-config-volume\") pod \"payload-processing-e2e-api-a-840559-65698dd65c-8d26g\" (UID: \"be58420c-fc67-4845-9088-65718c449087\") " pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:06:38.168541 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.168509 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be58420c-fc67-4845-9088-65718c449087-plugins-config-volume\") pod \"payload-processing-e2e-api-a-840559-65698dd65c-8d26g\" (UID: \"be58420c-fc67-4845-9088-65718c449087\") " pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:06:38.168735 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.168564 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pkpk\" (UniqueName: \"kubernetes.io/projected/be58420c-fc67-4845-9088-65718c449087-kube-api-access-4pkpk\") pod \"payload-processing-e2e-api-a-840559-65698dd65c-8d26g\" (UID: \"be58420c-fc67-4845-9088-65718c449087\") " pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:06:38.169212 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.169189 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be58420c-fc67-4845-9088-65718c449087-plugins-config-volume\") pod \"payload-processing-e2e-api-a-840559-65698dd65c-8d26g\" (UID: \"be58420c-fc67-4845-9088-65718c449087\") " pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:06:38.228299 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.228271 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pkpk\" (UniqueName: \"kubernetes.io/projected/be58420c-fc67-4845-9088-65718c449087-kube-api-access-4pkpk\") pod \"payload-processing-e2e-api-a-840559-65698dd65c-8d26g\" (UID: \"be58420c-fc67-4845-9088-65718c449087\") " pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:06:38.273948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.273920 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:06:38.283708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.283674 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" podUID="24d00911-860c-454b-b18d-4bed094c4d82" containerName="payload-processing" containerID="cri-o://cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1" gracePeriod=25 Nov 03 09:06:38.395255 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.395211 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" podUID="a55ec8f5-94f6-4c33-bc9e-36314df85e35" containerName="payload-pre-processing" containerID="cri-o://0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42" gracePeriod=25 Nov 03 09:06:38.664916 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.664883 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g"] Nov 03 09:06:38.665043 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:06:38.664956 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe58420c_fc67_4845_9088_65718c449087.slice/crio-5e00c91b2fdeb0b48870ea4073eed055024cdeca71f728887a0d920bae550393 WatchSource:0}: Error finding container 5e00c91b2fdeb0b48870ea4073eed055024cdeca71f728887a0d920bae550393: Status 404 returned error can't find the container with id 5e00c91b2fdeb0b48870ea4073eed055024cdeca71f728887a0d920bae550393 Nov 03 09:06:38.975864 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.975829 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" event={"ID":"be58420c-fc67-4845-9088-65718c449087","Type":"ContainerStarted","Data":"22aefdaeec7262c3409adb37981cf5774698cbb7a2c090774b31af28c6e6a695"} Nov 03 09:06:38.976272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.975890 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" event={"ID":"be58420c-fc67-4845-9088-65718c449087","Type":"ContainerStarted","Data":"5e00c91b2fdeb0b48870ea4073eed055024cdeca71f728887a0d920bae550393"} Nov 03 09:06:38.976272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:38.975916 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:06:39.093922 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:39.093862 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" podStartSLOduration=2.093844345 podStartE2EDuration="2.093844345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:06:39.09339601 +0000 UTC m=+1632.018338228" watchObservedRunningTime="2026-11-03 09:06:39.093844345 +0000 UTC m=+1632.018786564" Nov 03 09:06:40.008763 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:40.008729 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" podUID="a55ec8f5-94f6-4c33-bc9e-36314df85e35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:06:40.009183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:40.008767 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" podUID="24d00911-860c-454b-b18d-4bed094c4d82" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:06:42.999498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:42.998937 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8"] Nov 03 09:06:43.042365 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.041717 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8"] Nov 03 09:06:43.042365 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.041822 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:06:43.099590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.099566 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-04934e\"" Nov 03 09:06:43.149353 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.149327 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-04934e-dockercfg-xvjm5\"" Nov 03 09:06:43.155887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.155846 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g68tt\" (UniqueName: \"kubernetes.io/projected/6e38970c-7160-4875-86b9-90b48d1a0c9e-kube-api-access-g68tt\") pod \"payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8\" (UID: \"6e38970c-7160-4875-86b9-90b48d1a0c9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:06:43.156022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.155901 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e38970c-7160-4875-86b9-90b48d1a0c9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8\" (UID: \"6e38970c-7160-4875-86b9-90b48d1a0c9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:06:43.258205 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.258115 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g68tt\" (UniqueName: \"kubernetes.io/projected/6e38970c-7160-4875-86b9-90b48d1a0c9e-kube-api-access-g68tt\") pod \"payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8\" (UID: \"6e38970c-7160-4875-86b9-90b48d1a0c9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:06:43.258205 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.258175 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e38970c-7160-4875-86b9-90b48d1a0c9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8\" (UID: \"6e38970c-7160-4875-86b9-90b48d1a0c9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:06:43.259123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.259097 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e38970c-7160-4875-86b9-90b48d1a0c9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8\" (UID: \"6e38970c-7160-4875-86b9-90b48d1a0c9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:06:43.317983 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.317951 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g68tt\" (UniqueName: \"kubernetes.io/projected/6e38970c-7160-4875-86b9-90b48d1a0c9e-kube-api-access-g68tt\") pod \"payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8\" (UID: \"6e38970c-7160-4875-86b9-90b48d1a0c9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:06:43.357077 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.357046 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:06:43.744286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:43.744254 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8"] Nov 03 09:06:43.744712 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:06:43.744689 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e38970c_7160_4875_86b9_90b48d1a0c9e.slice/crio-e16891a42789c6c7bfd33015ef4e413816cd7382f02d0f97007a988a54c89f4f WatchSource:0}: Error finding container e16891a42789c6c7bfd33015ef4e413816cd7382f02d0f97007a988a54c89f4f: Status 404 returned error can't find the container with id e16891a42789c6c7bfd33015ef4e413816cd7382f02d0f97007a988a54c89f4f Nov 03 09:06:44.035063 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:44.034976 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" event={"ID":"6e38970c-7160-4875-86b9-90b48d1a0c9e","Type":"ContainerStarted","Data":"b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de"} Nov 03 09:06:44.035063 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:44.035033 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" event={"ID":"6e38970c-7160-4875-86b9-90b48d1a0c9e","Type":"ContainerStarted","Data":"e16891a42789c6c7bfd33015ef4e413816cd7382f02d0f97007a988a54c89f4f"} Nov 03 09:06:44.035492 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:44.035075 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:06:44.152756 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:44.152694 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" podStartSLOduration=2.152673779 podStartE2EDuration="2.152673779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:06:44.150357439 +0000 UTC m=+1637.075299681" watchObservedRunningTime="2026-11-03 09:06:44.152673779 +0000 UTC m=+1637.077615994" Nov 03 09:06:46.903142 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:06:46.903105 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:06:49.053655 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.053614 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2"] Nov 03 09:06:49.103501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.103445 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476a5c6-sdhh2"] Nov 03 09:06:49.206861 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.206823 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981df4b1-4b75-49e8-9df1-b74be2be6131" path="/var/lib/kubelet/pods/981df4b1-4b75-49e8-9df1-b74be2be6131/volumes" Nov 03 09:06:49.508328 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.508295 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5"] Nov 03 09:06:49.508553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.508533 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" podUID="2b8a4bcb-1e03-43d0-9785-8b3c15345594" containerName="maas-api" containerID="cri-o://480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654" gracePeriod=30 Nov 03 09:06:49.860500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.857091 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:06:49.977852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.977429 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b8a4bcb-1e03-43d0-9785-8b3c15345594-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8a4bcb-1e03-43d0-9785-8b3c15345594-openshift-service-ca\") pod \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " Nov 03 09:06:49.977852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.977713 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-maas-api-tls\") pod \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " Nov 03 09:06:49.977852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.977839 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8a4bcb-1e03-43d0-9785-8b3c15345594-kube-api-access-z2dsv\" (UniqueName: \"kubernetes.io/projected/2b8a4bcb-1e03-43d0-9785-8b3c15345594-kube-api-access-z2dsv\") pod \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " Nov 03 09:06:49.978162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.977912 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-metrics-certs\") pod \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\" (UID: \"2b8a4bcb-1e03-43d0-9785-8b3c15345594\") " Nov 03 09:06:49.982433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.979369 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8a4bcb-1e03-43d0-9785-8b3c15345594-openshift-service-ca" pod "2b8a4bcb-1e03-43d0-9785-8b3c15345594" (UID: "2b8a4bcb-1e03-43d0-9785-8b3c15345594"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:06:49.996190 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.995593 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-metrics-certs" pod "2b8a4bcb-1e03-43d0-9785-8b3c15345594" (UID: "2b8a4bcb-1e03-43d0-9785-8b3c15345594"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:06:49.996190 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.995775 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-maas-api-tls" pod "2b8a4bcb-1e03-43d0-9785-8b3c15345594" (UID: "2b8a4bcb-1e03-43d0-9785-8b3c15345594"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:06:49.997520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:49.997458 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8a4bcb-1e03-43d0-9785-8b3c15345594-kube-api-access-z2dsv" pod "2b8a4bcb-1e03-43d0-9785-8b3c15345594" (UID: "2b8a4bcb-1e03-43d0-9785-8b3c15345594"). InnerVolumeSpecName "kube-api-access-z2dsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:06:50.001309 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.001285 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:06:50.008821 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.008793 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" podUID="24d00911-860c-454b-b18d-4bed094c4d82" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:06:50.009611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.009538 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" podUID="a55ec8f5-94f6-4c33-bc9e-36314df85e35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:06:50.080242 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.080151 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8a4bcb-1e03-43d0-9785-8b3c15345594-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:50.080242 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.080181 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:50.080242 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.080191 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2dsv\" (UniqueName: \"kubernetes.io/projected/2b8a4bcb-1e03-43d0-9785-8b3c15345594-kube-api-access-z2dsv\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:50.080242 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.080200 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b8a4bcb-1e03-43d0-9785-8b3c15345594-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:50.089359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.089325 5015 generic.go:356] "Generic (PLEG): container finished" podID="2b8a4bcb-1e03-43d0-9785-8b3c15345594" containerID="480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654" exitCode=0 Nov 03 09:06:50.089572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.089369 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" Nov 03 09:06:50.089572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.089399 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" event={"ID":"2b8a4bcb-1e03-43d0-9785-8b3c15345594","Type":"ContainerDied","Data":"480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654"} Nov 03 09:06:50.089572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.089465 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5" event={"ID":"2b8a4bcb-1e03-43d0-9785-8b3c15345594","Type":"ContainerDied","Data":"38003ed03a813884ad879ca0f10176076019bfc75493e8b6090f388b52c294db"} Nov 03 09:06:50.089572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.089506 5015 scope.go:122] "RemoveContainer" containerID="480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654" Nov 03 09:06:50.104741 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.104716 5015 scope.go:122] "RemoveContainer" containerID="480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654" Nov 03 09:06:50.105117 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:06:50.105097 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654\": container with ID starting with 480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654 not found: ID does not exist" containerID="480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654" Nov 03 09:06:50.105198 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.105124 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654"} err="failed to get container status \"480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654\": rpc error: code = NotFound desc = could not find container \"480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654\": container with ID starting with 480d2965b5e62a01b5445a08cf026f0bf2298cbbc20e3fbc13738e8df0d84654 not found: ID does not exist" Nov 03 09:06:50.257026 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.256993 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5"] Nov 03 09:06:50.261159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:50.261137 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d7fd66-bbbdd8b65-c4zn5"] Nov 03 09:06:51.205771 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:51.205740 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8a4bcb-1e03-43d0-9785-8b3c15345594" path="/var/lib/kubelet/pods/2b8a4bcb-1e03-43d0-9785-8b3c15345594/volumes" Nov 03 09:06:53.761746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:53.761715 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97"] Nov 03 09:06:53.762139 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:53.761976 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" containerID="cri-o://4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc" gracePeriod=30 Nov 03 09:06:55.048967 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:55.048939 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:06:58.586617 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.586590 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:06:58.647037 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.646991 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d00911-860c-454b-b18d-4bed094c4d82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d00911-860c-454b-b18d-4bed094c4d82-plugins-config-volume\") pod \"24d00911-860c-454b-b18d-4bed094c4d82\" (UID: \"24d00911-860c-454b-b18d-4bed094c4d82\") " Nov 03 09:06:58.647228 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.647163 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d00911-860c-454b-b18d-4bed094c4d82-kube-api-access-tzkl4\" (UniqueName: \"kubernetes.io/projected/24d00911-860c-454b-b18d-4bed094c4d82-kube-api-access-tzkl4\") pod \"24d00911-860c-454b-b18d-4bed094c4d82\" (UID: \"24d00911-860c-454b-b18d-4bed094c4d82\") " Nov 03 09:06:58.648693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.648652 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d00911-860c-454b-b18d-4bed094c4d82-plugins-config-volume" pod "24d00911-860c-454b-b18d-4bed094c4d82" (UID: "24d00911-860c-454b-b18d-4bed094c4d82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:06:58.653010 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.652970 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d00911-860c-454b-b18d-4bed094c4d82-kube-api-access-tzkl4" pod "24d00911-860c-454b-b18d-4bed094c4d82" (UID: "24d00911-860c-454b-b18d-4bed094c4d82"). InnerVolumeSpecName "kube-api-access-tzkl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:06:58.687317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.687290 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:06:58.750027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.748713 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a55ec8f5-94f6-4c33-bc9e-36314df85e35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55ec8f5-94f6-4c33-bc9e-36314df85e35-plugins-config-volume\") pod \"a55ec8f5-94f6-4c33-bc9e-36314df85e35\" (UID: \"a55ec8f5-94f6-4c33-bc9e-36314df85e35\") " Nov 03 09:06:58.750027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.748794 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55ec8f5-94f6-4c33-bc9e-36314df85e35-kube-api-access-5nvtt\" (UniqueName: \"kubernetes.io/projected/a55ec8f5-94f6-4c33-bc9e-36314df85e35-kube-api-access-5nvtt\") pod \"a55ec8f5-94f6-4c33-bc9e-36314df85e35\" (UID: \"a55ec8f5-94f6-4c33-bc9e-36314df85e35\") " Nov 03 09:06:58.750027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.749652 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d00911-860c-454b-b18d-4bed094c4d82-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:58.750027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.749675 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzkl4\" (UniqueName: \"kubernetes.io/projected/24d00911-860c-454b-b18d-4bed094c4d82-kube-api-access-tzkl4\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:58.750027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.749920 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55ec8f5-94f6-4c33-bc9e-36314df85e35-plugins-config-volume" pod "a55ec8f5-94f6-4c33-bc9e-36314df85e35" (UID: "a55ec8f5-94f6-4c33-bc9e-36314df85e35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:06:58.752574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.752543 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55ec8f5-94f6-4c33-bc9e-36314df85e35-kube-api-access-5nvtt" pod "a55ec8f5-94f6-4c33-bc9e-36314df85e35" (UID: "a55ec8f5-94f6-4c33-bc9e-36314df85e35"). InnerVolumeSpecName "kube-api-access-5nvtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:06:58.851046 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.850961 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55ec8f5-94f6-4c33-bc9e-36314df85e35-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:58.851046 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:58.850993 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nvtt\" (UniqueName: \"kubernetes.io/projected/a55ec8f5-94f6-4c33-bc9e-36314df85e35-kube-api-access-5nvtt\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:59.031879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.031854 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:06:59.054092 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.054050 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg"] Nov 03 09:06:59.058235 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.058211 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c355d1-49vkg"] Nov 03 09:06:59.155838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.155748 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-kube-api-access-2cczs\" (UniqueName: \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-kube-api-access-2cczs\") pod \"52c2f4a9-66df-4c28-b236-d654ae8a3569\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " Nov 03 09:06:59.156025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.155923 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-credential-socket\") pod \"52c2f4a9-66df-4c28-b236-d654ae8a3569\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " Nov 03 09:06:59.156025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.155964 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c2f4a9-66df-4c28-b236-d654ae8a3569-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c2f4a9-66df-4c28-b236-d654ae8a3569-istiod-ca-cert\") pod \"52c2f4a9-66df-4c28-b236-d654ae8a3569\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " Nov 03 09:06:59.156025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.155996 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-socket\") pod \"52c2f4a9-66df-4c28-b236-d654ae8a3569\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " Nov 03 09:06:59.156224 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.156035 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-certs\") pod \"52c2f4a9-66df-4c28-b236-d654ae8a3569\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " Nov 03 09:06:59.156224 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.156180 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-data\") pod \"52c2f4a9-66df-4c28-b236-d654ae8a3569\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " Nov 03 09:06:59.156348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.156245 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-token\" (UniqueName: \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-token\") pod \"52c2f4a9-66df-4c28-b236-d654ae8a3569\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " Nov 03 09:06:59.156348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.156276 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-envoy\") pod \"52c2f4a9-66df-4c28-b236-d654ae8a3569\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " Nov 03 09:06:59.156451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.156357 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-podinfo\") pod \"52c2f4a9-66df-4c28-b236-d654ae8a3569\" (UID: \"52c2f4a9-66df-4c28-b236-d654ae8a3569\") " Nov 03 09:06:59.156807 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.156782 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-credential-socket" pod "52c2f4a9-66df-4c28-b236-d654ae8a3569" (UID: "52c2f4a9-66df-4c28-b236-d654ae8a3569"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:06:59.157079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.157062 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:59.157319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.157297 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-certs" pod "52c2f4a9-66df-4c28-b236-d654ae8a3569" (UID: "52c2f4a9-66df-4c28-b236-d654ae8a3569"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:06:59.157810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.157777 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c2f4a9-66df-4c28-b236-d654ae8a3569-istiod-ca-cert" pod "52c2f4a9-66df-4c28-b236-d654ae8a3569" (UID: "52c2f4a9-66df-4c28-b236-d654ae8a3569"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:06:59.158119 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.158083 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-data" pod "52c2f4a9-66df-4c28-b236-d654ae8a3569" (UID: "52c2f4a9-66df-4c28-b236-d654ae8a3569"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:06:59.158119 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.158099 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-socket" pod "52c2f4a9-66df-4c28-b236-d654ae8a3569" (UID: "52c2f4a9-66df-4c28-b236-d654ae8a3569"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:06:59.159855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.159826 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-podinfo" pod "52c2f4a9-66df-4c28-b236-d654ae8a3569" (UID: "52c2f4a9-66df-4c28-b236-d654ae8a3569"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:06:59.160292 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.160262 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-token" pod "52c2f4a9-66df-4c28-b236-d654ae8a3569" (UID: "52c2f4a9-66df-4c28-b236-d654ae8a3569"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:06:59.161655 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.161633 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-kube-api-access-2cczs" pod "52c2f4a9-66df-4c28-b236-d654ae8a3569" (UID: "52c2f4a9-66df-4c28-b236-d654ae8a3569"). InnerVolumeSpecName "kube-api-access-2cczs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:06:59.163817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.163793 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-envoy" pod "52c2f4a9-66df-4c28-b236-d654ae8a3569" (UID: "52c2f4a9-66df-4c28-b236-d654ae8a3569"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:06:59.172556 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.172525 5015 generic.go:356] "Generic (PLEG): container finished" podID="a55ec8f5-94f6-4c33-bc9e-36314df85e35" containerID="0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42" exitCode=0 Nov 03 09:06:59.172681 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.172663 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" event={"ID":"a55ec8f5-94f6-4c33-bc9e-36314df85e35","Type":"ContainerDied","Data":"0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42"} Nov 03 09:06:59.172746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.172710 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" event={"ID":"a55ec8f5-94f6-4c33-bc9e-36314df85e35","Type":"ContainerDied","Data":"d191459941fb98afd0c0d56b06989c9598375e80b75728687c714eeeacef8d73"} Nov 03 09:06:59.172746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.172743 5015 scope.go:122] "RemoveContainer" containerID="0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42" Nov 03 09:06:59.172913 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.172898 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf" Nov 03 09:06:59.179552 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.179265 5015 generic.go:356] "Generic (PLEG): container finished" podID="24d00911-860c-454b-b18d-4bed094c4d82" containerID="cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1" exitCode=0 Nov 03 09:06:59.179552 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.179458 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" event={"ID":"24d00911-860c-454b-b18d-4bed094c4d82","Type":"ContainerDied","Data":"cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1"} Nov 03 09:06:59.181236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.179766 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" event={"ID":"24d00911-860c-454b-b18d-4bed094c4d82","Type":"ContainerDied","Data":"52d2f7d0a1f30140eb0074f9431210e0541c554e95890fd148a536b8089fe6a5"} Nov 03 09:06:59.181236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.179863 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf" Nov 03 09:06:59.192521 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.191034 5015 generic.go:356] "Generic (PLEG): container finished" podID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerID="4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc" exitCode=0 Nov 03 09:06:59.192521 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.191201 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" event={"ID":"52c2f4a9-66df-4c28-b236-d654ae8a3569","Type":"ContainerDied","Data":"4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc"} Nov 03 09:06:59.192521 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.191246 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" event={"ID":"52c2f4a9-66df-4c28-b236-d654ae8a3569","Type":"ContainerDied","Data":"38725252b01bd5972b19488ad629855ef4bb5c5f46eec3dd5c1f5efeba5aa9bc"} Nov 03 09:06:59.192521 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.191319 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97" Nov 03 09:06:59.195039 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.195019 5015 scope.go:122] "RemoveContainer" containerID="0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42" Nov 03 09:06:59.195314 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:06:59.195294 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42\": container with ID starting with 0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42 not found: ID does not exist" containerID="0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42" Nov 03 09:06:59.195397 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.195320 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42"} err="failed to get container status \"0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42\": rpc error: code = NotFound desc = could not find container \"0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42\": container with ID starting with 0ac056131cb964de6728a88697f6c65b167dccee5125c8c203cc887733fabc42 not found: ID does not exist" Nov 03 09:06:59.195397 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.195339 5015 scope.go:122] "RemoveContainer" containerID="cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1" Nov 03 09:06:59.207106 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.207078 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f85603a-e596-4a81-a6e9-d843f4be7901" path="/var/lib/kubelet/pods/2f85603a-e596-4a81-a6e9-d843f4be7901/volumes" Nov 03 09:06:59.210101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.209912 5015 scope.go:122] "RemoveContainer" containerID="cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1" Nov 03 09:06:59.210439 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:06:59.210415 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1\": container with ID starting with cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1 not found: ID does not exist" containerID="cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1" Nov 03 09:06:59.210578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.210446 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1"} err="failed to get container status \"cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1\": rpc error: code = NotFound desc = could not find container \"cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1\": container with ID starting with cb01cef461b96432ec01bbbcadac46b26f340a56c02ec0b65069bba64cdde3a1 not found: ID does not exist" Nov 03 09:06:59.210578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.210464 5015 scope.go:122] "RemoveContainer" containerID="4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc" Nov 03 09:06:59.223016 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.222988 5015 scope.go:122] "RemoveContainer" containerID="7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969" Nov 03 09:06:59.244986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.244961 5015 scope.go:122] "RemoveContainer" containerID="4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc" Nov 03 09:06:59.245299 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:06:59.245277 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc\": container with ID starting with 4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc not found: ID does not exist" containerID="4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc" Nov 03 09:06:59.245361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.245308 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc"} err="failed to get container status \"4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc\": rpc error: code = NotFound desc = could not find container \"4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc\": container with ID starting with 4f2913c3bfb1a75ace2a1fca8af4e1edf7983335c2ad664a093c6dbf80f3eafc not found: ID does not exist" Nov 03 09:06:59.245361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.245326 5015 scope.go:122] "RemoveContainer" containerID="7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969" Nov 03 09:06:59.245656 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:06:59.245634 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969\": container with ID starting with 7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969 not found: ID does not exist" containerID="7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969" Nov 03 09:06:59.245714 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.245666 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969"} err="failed to get container status \"7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969\": rpc error: code = NotFound desc = could not find container \"7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969\": container with ID starting with 7aa98983247ad0b543a982733ccf995ae83e537f62e64f03d8b30ba1379a4969 not found: ID does not exist" Nov 03 09:06:59.258687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.258664 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:59.258784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.258690 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:59.258784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.258699 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:59.258784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.258708 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:59.258784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.258716 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:59.258784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.258724 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c2f4a9-66df-4c28-b236-d654ae8a3569-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:59.258784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.258732 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cczs\" (UniqueName: \"kubernetes.io/projected/52c2f4a9-66df-4c28-b236-d654ae8a3569-kube-api-access-2cczs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:59.258784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.258742 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c2f4a9-66df-4c28-b236-d654ae8a3569-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:06:59.353232 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.352453 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97"] Nov 03 09:06:59.356536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.356515 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d7fd66-openshift-default-8576fd6887-sns97"] Nov 03 09:06:59.467005 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.466829 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf"] Nov 03 09:06:59.471594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.471566 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-wdnhf"] Nov 03 09:06:59.587145 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.587101 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf"] Nov 03 09:06:59.591054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:06:59.591017 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-dd5mf"] Nov 03 09:07:00.212560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:00.212530 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh"] Nov 03 09:07:01.206018 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.205989 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d00911-860c-454b-b18d-4bed094c4d82" path="/var/lib/kubelet/pods/24d00911-860c-454b-b18d-4bed094c4d82/volumes" Nov 03 09:07:01.206526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.206510 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" path="/var/lib/kubelet/pods/52c2f4a9-66df-4c28-b236-d654ae8a3569/volumes" Nov 03 09:07:01.206902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.206889 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a55ec8f5-94f6-4c33-bc9e-36314df85e35" path="/var/lib/kubelet/pods/a55ec8f5-94f6-4c33-bc9e-36314df85e35/volumes" Nov 03 09:07:01.831048 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831014 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9"] Nov 03 09:07:01.831712 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831688 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a55ec8f5-94f6-4c33-bc9e-36314df85e35" containerName="payload-pre-processing" Nov 03 09:07:01.831712 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831704 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55ec8f5-94f6-4c33-bc9e-36314df85e35" containerName="payload-pre-processing" Nov 03 09:07:01.831799 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831726 5015 cpu_manager.go:414] "containerMap: removing container" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" Nov 03 09:07:01.831799 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831731 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" Nov 03 09:07:01.831799 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831762 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2b8a4bcb-1e03-43d0-9785-8b3c15345594" containerName="maas-api" Nov 03 09:07:01.831799 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831769 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8a4bcb-1e03-43d0-9785-8b3c15345594" containerName="maas-api" Nov 03 09:07:01.831975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831812 5015 cpu_manager.go:414] "containerMap: removing container" podUID="24d00911-860c-454b-b18d-4bed094c4d82" containerName="payload-processing" Nov 03 09:07:01.831975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831820 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d00911-860c-454b-b18d-4bed094c4d82" containerName="payload-processing" Nov 03 09:07:01.831975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831959 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55ec8f5-94f6-4c33-bc9e-36314df85e35" containerName="payload-pre-processing" Nov 03 09:07:01.831975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831969 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" Nov 03 09:07:01.832119 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831979 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8a4bcb-1e03-43d0-9785-8b3c15345594" containerName="maas-api" Nov 03 09:07:01.832119 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.831991 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d00911-860c-454b-b18d-4bed094c4d82" containerName="payload-processing" Nov 03 09:07:01.837614 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.837592 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:01.900978 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.900942 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhpx\" (UniqueName: \"kubernetes.io/projected/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-kube-api-access-vrhpx\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:01.901634 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.901356 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-metrics-certs\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:01.901920 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.901899 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-openshift-service-ca\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:01.902019 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.901957 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-maas-api-tls\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:01.949129 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.949086 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9"] Nov 03 09:07:01.949326 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:01.949173 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-eb9cd2-serving-cert\"" Nov 03 09:07:02.003170 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:02.003138 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhpx\" (UniqueName: \"kubernetes.io/projected/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-kube-api-access-vrhpx\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:02.003357 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:02.003293 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-metrics-certs\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:02.003357 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:02.003334 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-openshift-service-ca\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:02.003489 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:02.003378 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-maas-api-tls\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:02.003583 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:07:02.003561 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-serving-cert: secret "maas-api-e2e-api-b-eb9cd2-serving-cert" not found Nov 03 09:07:02.003641 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:07:02.003628 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-maas-api-tls podName:c29c5af7-5cf9-4865-a962-e7ce5f5a8755 nodeName:}" failed. No retries permitted until 2026-11-03 09:07:02.503610484 +0000 UTC m=+1655.428552706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-maas-api-tls") pod "maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" (UID: "c29c5af7-5cf9-4865-a962-e7ce5f5a8755") : secret "maas-api-e2e-api-b-eb9cd2-serving-cert" not found Nov 03 09:07:02.004101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:02.004082 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-openshift-service-ca\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:02.006129 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:02.006099 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-metrics-certs\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:02.064621 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:02.064588 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhpx\" (UniqueName: \"kubernetes.io/projected/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-kube-api-access-vrhpx\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:02.511797 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:02.511765 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-maas-api-tls\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:02.514453 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:02.514433 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-maas-api-tls\") pod \"maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:02.753918 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:02.753883 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:03.138985 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:07:03.138947 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29c5af7_5cf9_4865_a962_e7ce5f5a8755.slice/crio-fa066e3df85985736508a07c1a5475def3ba65bf64edc46e0105cdbc9cdd43e7 WatchSource:0}: Error finding container fa066e3df85985736508a07c1a5475def3ba65bf64edc46e0105cdbc9cdd43e7: Status 404 returned error can't find the container with id fa066e3df85985736508a07c1a5475def3ba65bf64edc46e0105cdbc9cdd43e7 Nov 03 09:07:03.139754 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:03.139588 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9"] Nov 03 09:07:03.236720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:03.236686 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" event={"ID":"c29c5af7-5cf9-4865-a962-e7ce5f5a8755","Type":"ContainerStarted","Data":"fa066e3df85985736508a07c1a5475def3ba65bf64edc46e0105cdbc9cdd43e7"} Nov 03 09:07:04.247444 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:04.247411 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" event={"ID":"c29c5af7-5cf9-4865-a962-e7ce5f5a8755","Type":"ContainerStarted","Data":"a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c"} Nov 03 09:07:04.247865 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:04.247663 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:04.388132 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:04.387956 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" podStartSLOduration=2.402886081 podStartE2EDuration="3.387936683s" totalImagesPullingTime="985.050602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:07:04.387627365 +0000 UTC m=+1657.312569615" watchObservedRunningTime="2026-11-03 09:07:04.387936683 +0000 UTC m=+1657.312878902" Nov 03 09:07:05.212943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:05.212857 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" podUID="df586e68-525d-4a8a-9926-fe81845c9be9" containerName="payload-pre-processing" containerID="cri-o://098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9" gracePeriod=25 Nov 03 09:07:05.595794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:05.595765 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:07:05.672566 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:05.672533 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df586e68-525d-4a8a-9926-fe81845c9be9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df586e68-525d-4a8a-9926-fe81845c9be9-plugins-config-volume\") pod \"df586e68-525d-4a8a-9926-fe81845c9be9\" (UID: \"df586e68-525d-4a8a-9926-fe81845c9be9\") " Nov 03 09:07:05.672757 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:05.672658 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df586e68-525d-4a8a-9926-fe81845c9be9-kube-api-access-jn9gz\" (UniqueName: \"kubernetes.io/projected/df586e68-525d-4a8a-9926-fe81845c9be9-kube-api-access-jn9gz\") pod \"df586e68-525d-4a8a-9926-fe81845c9be9\" (UID: \"df586e68-525d-4a8a-9926-fe81845c9be9\") " Nov 03 09:07:05.672916 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:05.672893 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df586e68-525d-4a8a-9926-fe81845c9be9-plugins-config-volume" pod "df586e68-525d-4a8a-9926-fe81845c9be9" (UID: "df586e68-525d-4a8a-9926-fe81845c9be9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:07:05.673317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:05.673294 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df586e68-525d-4a8a-9926-fe81845c9be9-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:05.675047 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:05.675018 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df586e68-525d-4a8a-9926-fe81845c9be9-kube-api-access-jn9gz" pod "df586e68-525d-4a8a-9926-fe81845c9be9" (UID: "df586e68-525d-4a8a-9926-fe81845c9be9"). InnerVolumeSpecName "kube-api-access-jn9gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:07:05.775082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:05.775045 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn9gz\" (UniqueName: \"kubernetes.io/projected/df586e68-525d-4a8a-9926-fe81845c9be9-kube-api-access-jn9gz\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:06.267554 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:06.267518 5015 generic.go:356] "Generic (PLEG): container finished" podID="df586e68-525d-4a8a-9926-fe81845c9be9" containerID="098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9" exitCode=0 Nov 03 09:07:06.267768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:06.267560 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" Nov 03 09:07:06.267768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:06.267601 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" event={"ID":"df586e68-525d-4a8a-9926-fe81845c9be9","Type":"ContainerDied","Data":"098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9"} Nov 03 09:07:06.267768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:06.267664 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh" event={"ID":"df586e68-525d-4a8a-9926-fe81845c9be9","Type":"ContainerDied","Data":"18c544a7c321dd8b2c81334c9254a4e74f057991cecbb61cd86ed8f7d05621ba"} Nov 03 09:07:06.267768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:06.267692 5015 scope.go:122] "RemoveContainer" containerID="098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9" Nov 03 09:07:06.280211 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:06.280194 5015 scope.go:122] "RemoveContainer" containerID="098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9" Nov 03 09:07:06.280522 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:07:06.280495 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9\": container with ID starting with 098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9 not found: ID does not exist" containerID="098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9" Nov 03 09:07:06.280610 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:06.280535 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9"} err="failed to get container status \"098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9\": rpc error: code = NotFound desc = could not find container \"098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9\": container with ID starting with 098b2c5ba2972b3ae3b69a1800469a42d63ad7d339d043fc884348fa345837d9 not found: ID does not exist" Nov 03 09:07:06.440194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:06.440158 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh"] Nov 03 09:07:06.489857 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:06.489821 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b8fd34-66fcf95d67-l77vh"] Nov 03 09:07:07.205861 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:07.205824 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df586e68-525d-4a8a-9926-fe81845c9be9" path="/var/lib/kubelet/pods/df586e68-525d-4a8a-9926-fe81845c9be9/volumes" Nov 03 09:07:08.266547 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:08.266504 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg"] Nov 03 09:07:08.267244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:08.266735 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerName="istio-proxy" containerID="cri-o://336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41" gracePeriod=30 Nov 03 09:07:10.261920 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:10.261893 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:07:13.564143 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.564112 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:07:13.580407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.580380 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-credential-socket\") pod \"e3b52cce-6255-4264-af07-e18a4352eb5a\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " Nov 03 09:07:13.580572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.580447 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-certs\") pod \"e3b52cce-6255-4264-af07-e18a4352eb5a\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " Nov 03 09:07:13.580572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.580499 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-kube-api-access-l2h2l\" (UniqueName: \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-kube-api-access-l2h2l\") pod \"e3b52cce-6255-4264-af07-e18a4352eb5a\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " Nov 03 09:07:13.580572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.580553 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-data\") pod \"e3b52cce-6255-4264-af07-e18a4352eb5a\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " Nov 03 09:07:13.580757 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.580629 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-token\" (UniqueName: \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-token\") pod \"e3b52cce-6255-4264-af07-e18a4352eb5a\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " Nov 03 09:07:13.580757 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.580641 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-credential-socket" pod "e3b52cce-6255-4264-af07-e18a4352eb5a" (UID: "e3b52cce-6255-4264-af07-e18a4352eb5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:07:13.580757 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.580731 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-envoy\") pod \"e3b52cce-6255-4264-af07-e18a4352eb5a\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " Nov 03 09:07:13.580908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.580765 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b52cce-6255-4264-af07-e18a4352eb5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b52cce-6255-4264-af07-e18a4352eb5a-istiod-ca-cert\") pod \"e3b52cce-6255-4264-af07-e18a4352eb5a\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " Nov 03 09:07:13.580908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.580781 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-certs" pod "e3b52cce-6255-4264-af07-e18a4352eb5a" (UID: "e3b52cce-6255-4264-af07-e18a4352eb5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:07:13.580908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.580801 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-socket\") pod \"e3b52cce-6255-4264-af07-e18a4352eb5a\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " Nov 03 09:07:13.580908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.580860 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-podinfo\") pod \"e3b52cce-6255-4264-af07-e18a4352eb5a\" (UID: \"e3b52cce-6255-4264-af07-e18a4352eb5a\") " Nov 03 09:07:13.581122 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.581071 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-socket" pod "e3b52cce-6255-4264-af07-e18a4352eb5a" (UID: "e3b52cce-6255-4264-af07-e18a4352eb5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:07:13.581182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.581139 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-data" pod "e3b52cce-6255-4264-af07-e18a4352eb5a" (UID: "e3b52cce-6255-4264-af07-e18a4352eb5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:07:13.581182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.581171 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b52cce-6255-4264-af07-e18a4352eb5a-istiod-ca-cert" pod "e3b52cce-6255-4264-af07-e18a4352eb5a" (UID: "e3b52cce-6255-4264-af07-e18a4352eb5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:07:13.581603 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.581584 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b52cce-6255-4264-af07-e18a4352eb5a-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.581686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.581605 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.581686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.581619 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.581686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.581631 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.581686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.581643 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.583035 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.583014 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-token" pod "e3b52cce-6255-4264-af07-e18a4352eb5a" (UID: "e3b52cce-6255-4264-af07-e18a4352eb5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:07:13.583131 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.583075 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-kube-api-access-l2h2l" pod "e3b52cce-6255-4264-af07-e18a4352eb5a" (UID: "e3b52cce-6255-4264-af07-e18a4352eb5a"). InnerVolumeSpecName "kube-api-access-l2h2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:07:13.583538 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.583521 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-envoy" pod "e3b52cce-6255-4264-af07-e18a4352eb5a" (UID: "e3b52cce-6255-4264-af07-e18a4352eb5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:07:13.583623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.583603 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-podinfo" pod "e3b52cce-6255-4264-af07-e18a4352eb5a" (UID: "e3b52cce-6255-4264-af07-e18a4352eb5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:07:13.682864 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.682821 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2h2l\" (UniqueName: \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-kube-api-access-l2h2l\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.682864 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.682856 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.682864 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.682869 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.683098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:13.682881 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b52cce-6255-4264-af07-e18a4352eb5a-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:14.340366 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:14.340328 5015 generic.go:356] "Generic (PLEG): container finished" podID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerID="336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41" exitCode=0 Nov 03 09:07:14.340604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:14.340371 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" Nov 03 09:07:14.340604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:14.340410 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" event={"ID":"e3b52cce-6255-4264-af07-e18a4352eb5a","Type":"ContainerDied","Data":"336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41"} Nov 03 09:07:14.340604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:14.340510 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg" event={"ID":"e3b52cce-6255-4264-af07-e18a4352eb5a","Type":"ContainerDied","Data":"70161227d5193c791d27ece685f994ed6800052f6bce0b7ad32014a3cfc0bacb"} Nov 03 09:07:14.340604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:14.340544 5015 scope.go:122] "RemoveContainer" containerID="336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41" Nov 03 09:07:14.353743 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:14.353724 5015 scope.go:122] "RemoveContainer" containerID="336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41" Nov 03 09:07:14.354088 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:07:14.354066 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41\": container with ID starting with 336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41 not found: ID does not exist" containerID="336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41" Nov 03 09:07:14.354179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:14.354092 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41"} err="failed to get container status \"336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41\": rpc error: code = NotFound desc = could not find container \"336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41\": container with ID starting with 336e78b21b194b3ad0253cf4253c4b1bf906421740db314712ead63a5265be41 not found: ID does not exist" Nov 03 09:07:14.562866 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:14.562831 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg"] Nov 03 09:07:14.613540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:14.613445 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-k42mg"] Nov 03 09:07:15.206202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:15.206167 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" path="/var/lib/kubelet/pods/e3b52cce-6255-4264-af07-e18a4352eb5a/volumes" Nov 03 09:07:16.902552 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:07:16.902513 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:07:17.055738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:17.055701 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz"] Nov 03 09:07:17.059437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:17.059409 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49d332e-tdgfz"] Nov 03 09:07:17.205461 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:17.205381 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1e693b-9d3e-4082-91e9-ee5961cadc17" path="/var/lib/kubelet/pods/2a1e693b-9d3e-4082-91e9-ee5961cadc17/volumes" Nov 03 09:07:23.884896 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.884857 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv"] Nov 03 09:07:23.885770 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.885752 5015 cpu_manager.go:414] "containerMap: removing container" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerName="istio-proxy" Nov 03 09:07:23.885829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.885771 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerName="istio-proxy" Nov 03 09:07:23.885829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.885784 5015 cpu_manager.go:414] "containerMap: removing container" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" Nov 03 09:07:23.885829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.885789 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" Nov 03 09:07:23.885829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.885800 5015 cpu_manager.go:414] "containerMap: removing container" podUID="df586e68-525d-4a8a-9926-fe81845c9be9" containerName="payload-pre-processing" Nov 03 09:07:23.885829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.885805 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df586e68-525d-4a8a-9926-fe81845c9be9" containerName="payload-pre-processing" Nov 03 09:07:23.885985 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.885926 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c2f4a9-66df-4c28-b236-d654ae8a3569" containerName="istio-proxy" Nov 03 09:07:23.885985 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.885935 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="df586e68-525d-4a8a-9926-fe81845c9be9" containerName="payload-pre-processing" Nov 03 09:07:23.885985 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.885951 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b52cce-6255-4264-af07-e18a4352eb5a" containerName="istio-proxy" Nov 03 09:07:23.890794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.890776 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:23.947891 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.947858 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv"] Nov 03 09:07:23.991216 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:23.991186 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-840559-serving-cert\"" Nov 03 09:07:24.030068 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.030032 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-metrics-certs\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.030259 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.030112 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064c9fc1-8346-4d21-9301-5becefffa0c5-openshift-service-ca\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.030259 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.030167 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-maas-api-tls\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.030259 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.030250 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7krzp\" (UniqueName: \"kubernetes.io/projected/064c9fc1-8346-4d21-9301-5becefffa0c5-kube-api-access-7krzp\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.131751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.131720 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-metrics-certs\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.131955 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.131797 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064c9fc1-8346-4d21-9301-5becefffa0c5-openshift-service-ca\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.131955 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.131828 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-maas-api-tls\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.132089 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.132032 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krzp\" (UniqueName: \"kubernetes.io/projected/064c9fc1-8346-4d21-9301-5becefffa0c5-kube-api-access-7krzp\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.132691 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:07:24.132668 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-840559-serving-cert: secret "maas-api-e2e-api-a-840559-serving-cert" not found Nov 03 09:07:24.132914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.132698 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064c9fc1-8346-4d21-9301-5becefffa0c5-openshift-service-ca\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.132914 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:07:24.132774 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-maas-api-tls podName:064c9fc1-8346-4d21-9301-5becefffa0c5 nodeName:}" failed. No retries permitted until 2026-11-03 09:07:24.632752102 +0000 UTC m=+1677.557694315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-maas-api-tls") pod "maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" (UID: "064c9fc1-8346-4d21-9301-5becefffa0c5") : secret "maas-api-e2e-api-a-840559-serving-cert" not found Nov 03 09:07:24.136617 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.136556 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-metrics-certs\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.194366 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.194320 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krzp\" (UniqueName: \"kubernetes.io/projected/064c9fc1-8346-4d21-9301-5becefffa0c5-kube-api-access-7krzp\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.640203 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.640163 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-maas-api-tls\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.642865 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.642836 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-maas-api-tls\") pod \"maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:24.805551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:24.805518 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:25.195953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:25.195923 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv"] Nov 03 09:07:25.196295 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:07:25.195959 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064c9fc1_8346_4d21_9301_5becefffa0c5.slice/crio-b0c9d3e8cf33986d42e4b89af747a5a802bdb9ac347191ac9468b85c8ec25d5c WatchSource:0}: Error finding container b0c9d3e8cf33986d42e4b89af747a5a802bdb9ac347191ac9468b85c8ec25d5c: Status 404 returned error can't find the container with id b0c9d3e8cf33986d42e4b89af747a5a802bdb9ac347191ac9468b85c8ec25d5c Nov 03 09:07:25.439715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:25.439677 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" event={"ID":"064c9fc1-8346-4d21-9301-5becefffa0c5","Type":"ContainerStarted","Data":"b0c9d3e8cf33986d42e4b89af747a5a802bdb9ac347191ac9468b85c8ec25d5c"} Nov 03 09:07:26.457886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:26.455959 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" event={"ID":"064c9fc1-8346-4d21-9301-5becefffa0c5","Type":"ContainerStarted","Data":"d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a"} Nov 03 09:07:26.457886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:26.456246 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:26.580022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:26.579957 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" podStartSLOduration=2.793263044 podStartE2EDuration="3.57993803s" totalImagesPullingTime="786.674986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:07:26.578570799 +0000 UTC m=+1679.503513018" watchObservedRunningTime="2026-11-03 09:07:26.57993803 +0000 UTC m=+1679.504880249" Nov 03 09:07:27.628368 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:27.628337 5015 scope.go:122] "RemoveContainer" containerID="bc95a089bf6211f59b7e0ac61baaa2efd07c21514c6a9806b1ea299f830546d4" Nov 03 09:07:28.056756 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:28.056719 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b"] Nov 03 09:07:28.104329 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:28.104298 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6b4aee-5xj5b"] Nov 03 09:07:29.206181 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:29.206150 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651dfc4a-7ddd-49e9-ae18-386a259022ac" path="/var/lib/kubelet/pods/651dfc4a-7ddd-49e9-ae18-386a259022ac/volumes" Nov 03 09:07:32.482215 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:32.482188 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:07:46.904246 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:07:46.904207 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:07:54.355633 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.355597 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd"] Nov 03 09:07:54.356167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.355919 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" podUID="cac04ca7-4fa9-4350-878f-5716bbde17ba" containerName="maas-api" containerID="cri-o://f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228" gracePeriod=30 Nov 03 09:07:54.696358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.696329 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:07:54.716687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.716651 5015 generic.go:356] "Generic (PLEG): container finished" podID="cac04ca7-4fa9-4350-878f-5716bbde17ba" containerID="f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228" exitCode=0 Nov 03 09:07:54.716859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.716707 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" Nov 03 09:07:54.716859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.716733 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" event={"ID":"cac04ca7-4fa9-4350-878f-5716bbde17ba","Type":"ContainerDied","Data":"f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228"} Nov 03 09:07:54.716859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.716795 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd" event={"ID":"cac04ca7-4fa9-4350-878f-5716bbde17ba","Type":"ContainerDied","Data":"a39329b8f963c314b73830f5bf70f18073950d2793117a0d6a1fa64f0be50646"} Nov 03 09:07:54.716859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.716823 5015 scope.go:122] "RemoveContainer" containerID="f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228" Nov 03 09:07:54.727262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.727246 5015 scope.go:122] "RemoveContainer" containerID="c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9" Nov 03 09:07:54.736696 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.736673 5015 scope.go:122] "RemoveContainer" containerID="f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228" Nov 03 09:07:54.737016 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:07:54.736993 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228\": container with ID starting with f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228 not found: ID does not exist" containerID="f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228" Nov 03 09:07:54.737109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.737025 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228"} err="failed to get container status \"f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228\": rpc error: code = NotFound desc = could not find container \"f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228\": container with ID starting with f829569dda4e30f95bc4af422f0b289ac506bf62cc590684a91fd8860698d228 not found: ID does not exist" Nov 03 09:07:54.737109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.737066 5015 scope.go:122] "RemoveContainer" containerID="c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9" Nov 03 09:07:54.737331 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:07:54.737310 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9\": container with ID starting with c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9 not found: ID does not exist" containerID="c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9" Nov 03 09:07:54.737401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.737340 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9"} err="failed to get container status \"c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9\": rpc error: code = NotFound desc = could not find container \"c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9\": container with ID starting with c4703fa4d382175c9b0b1f8b6776e0a408dc0ee33414f4c2567a6af9143aacc9 not found: ID does not exist" Nov 03 09:07:54.814666 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.814633 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-maas-api-tls\") pod \"cac04ca7-4fa9-4350-878f-5716bbde17ba\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " Nov 03 09:07:54.814889 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.814786 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac04ca7-4fa9-4350-878f-5716bbde17ba-kube-api-access-x9lgq\" (UniqueName: \"kubernetes.io/projected/cac04ca7-4fa9-4350-878f-5716bbde17ba-kube-api-access-x9lgq\") pod \"cac04ca7-4fa9-4350-878f-5716bbde17ba\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " Nov 03 09:07:54.814963 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.814925 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac04ca7-4fa9-4350-878f-5716bbde17ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac04ca7-4fa9-4350-878f-5716bbde17ba-openshift-service-ca\") pod \"cac04ca7-4fa9-4350-878f-5716bbde17ba\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " Nov 03 09:07:54.815856 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.815028 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-metrics-certs\") pod \"cac04ca7-4fa9-4350-878f-5716bbde17ba\" (UID: \"cac04ca7-4fa9-4350-878f-5716bbde17ba\") " Nov 03 09:07:54.815856 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.815415 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac04ca7-4fa9-4350-878f-5716bbde17ba-openshift-service-ca" pod "cac04ca7-4fa9-4350-878f-5716bbde17ba" (UID: "cac04ca7-4fa9-4350-878f-5716bbde17ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:07:54.815856 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.815816 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac04ca7-4fa9-4350-878f-5716bbde17ba-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:54.820577 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.818457 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-maas-api-tls" pod "cac04ca7-4fa9-4350-878f-5716bbde17ba" (UID: "cac04ca7-4fa9-4350-878f-5716bbde17ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:07:54.820577 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.818654 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-metrics-certs" pod "cac04ca7-4fa9-4350-878f-5716bbde17ba" (UID: "cac04ca7-4fa9-4350-878f-5716bbde17ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:07:54.820577 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.819734 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac04ca7-4fa9-4350-878f-5716bbde17ba-kube-api-access-x9lgq" pod "cac04ca7-4fa9-4350-878f-5716bbde17ba" (UID: "cac04ca7-4fa9-4350-878f-5716bbde17ba"). InnerVolumeSpecName "kube-api-access-x9lgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:07:54.917410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.917332 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9lgq\" (UniqueName: \"kubernetes.io/projected/cac04ca7-4fa9-4350-878f-5716bbde17ba-kube-api-access-x9lgq\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:54.917410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.917361 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:54.917410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:54.917371 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac04ca7-4fa9-4350-878f-5716bbde17ba-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:55.147776 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:55.147741 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd"] Nov 03 09:07:55.153947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:55.153918 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8fd34-c6799dfc4-hn7hd"] Nov 03 09:07:55.207194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:07:55.207120 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac04ca7-4fa9-4350-878f-5716bbde17ba" path="/var/lib/kubelet/pods/cac04ca7-4fa9-4350-878f-5716bbde17ba/volumes" Nov 03 09:08:14.301831 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:14.301794 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld"] Nov 03 09:08:14.414882 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:14.414827 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb"] Nov 03 09:08:16.905538 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:08:16.905459 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:08:17.659049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.658996 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr"] Nov 03 09:08:17.680303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.680267 5015 cpu_manager.go:414] "containerMap: removing container" podUID="cac04ca7-4fa9-4350-878f-5716bbde17ba" containerName="maas-api" Nov 03 09:08:17.680303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.680301 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac04ca7-4fa9-4350-878f-5716bbde17ba" containerName="maas-api" Nov 03 09:08:17.680580 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.680327 5015 cpu_manager.go:414] "containerMap: removing container" podUID="cac04ca7-4fa9-4350-878f-5716bbde17ba" containerName="maas-api" Nov 03 09:08:17.680580 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.680335 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac04ca7-4fa9-4350-878f-5716bbde17ba" containerName="maas-api" Nov 03 09:08:17.680580 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.680555 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac04ca7-4fa9-4350-878f-5716bbde17ba" containerName="maas-api" Nov 03 09:08:17.680580 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.680570 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac04ca7-4fa9-4350-878f-5716bbde17ba" containerName="maas-api" Nov 03 09:08:17.686778 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.686752 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.778324 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.778284 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr"] Nov 03 09:08:17.792387 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.792354 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4ae3db-openshift-default-dockercfg-psczd\"" Nov 03 09:08:17.822184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.817523 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-podinfo\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.822184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.817624 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istiod-ca-cert\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.822184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.817682 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g462s\" (UniqueName: \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-kube-api-access-g462s\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.822184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.817761 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-socket\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.822184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.817880 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-token\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.822184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.817919 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-data\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.822184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.817998 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-credential-socket\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.822184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.818035 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-certs\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.822184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.818065 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-envoy\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.919692 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.919598 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g462s\" (UniqueName: \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-kube-api-access-g462s\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.920174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.919709 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-socket\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.920174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.919820 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-token\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.920174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.919854 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-data\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.920174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.919930 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-credential-socket\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.920174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.919965 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-certs\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.920174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.919993 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-envoy\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.920174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.920090 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-podinfo\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.920174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.920168 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istiod-ca-cert\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.920693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.920585 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-socket\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.920976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.920948 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-certs\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.921139 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.921041 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istiod-ca-cert\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.921227 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.921208 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-data\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.921500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.921459 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-credential-socket\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.925817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.925792 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-envoy\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.926201 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.926179 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-podinfo\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.982251 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.980233 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g462s\" (UniqueName: \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-kube-api-access-g462s\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:17.982251 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:17.980912 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-token\") pod \"e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:18.016219 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:18.015896 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:18.536022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:18.535988 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr"] Nov 03 09:08:18.536950 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:08:18.536920 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d1b378_6be0_4b3d_95ef_71b82d571d7f.slice/crio-f9db319a77154d9a9071ea8adb784662abc2606a9b4307ea8db055eacb9b14e9 WatchSource:0}: Error finding container f9db319a77154d9a9071ea8adb784662abc2606a9b4307ea8db055eacb9b14e9: Status 404 returned error can't find the container with id f9db319a77154d9a9071ea8adb784662abc2606a9b4307ea8db055eacb9b14e9 Nov 03 09:08:18.540295 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:18.540248 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:08:18.540409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:18.540347 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:08:18.540409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:18.540397 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:08:19.000005 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:18.999972 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" event={"ID":"11d1b378-6be0-4b3d-95ef-71b82d571d7f","Type":"ContainerStarted","Data":"389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49"} Nov 03 09:08:19.000520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.000045 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" event={"ID":"11d1b378-6be0-4b3d-95ef-71b82d571d7f","Type":"ContainerStarted","Data":"f9db319a77154d9a9071ea8adb784662abc2606a9b4307ea8db055eacb9b14e9"} Nov 03 09:08:19.017683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.017658 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:19.023590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.023567 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:19.120149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.120100 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" podStartSLOduration=2.120072465 podStartE2EDuration="2.120072465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:08:19.117789958 +0000 UTC m=+1732.042732217" watchObservedRunningTime="2026-11-03 09:08:19.120072465 +0000 UTC m=+1732.045014719" Nov 03 09:08:19.302702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.302579 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" podUID="4f9c75c1-ec86-4637-be65-48efdda20ba7" containerName="payload-processing" containerID="cri-o://89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21" gracePeriod=25 Nov 03 09:08:19.415390 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.415329 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" podUID="97c50c64-e1d9-4f70-bf80-12713de54c41" containerName="payload-pre-processing" containerID="cri-o://22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2" gracePeriod=25 Nov 03 09:08:19.634511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.630813 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:08:19.738581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.738534 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9c75c1-ec86-4637-be65-48efdda20ba7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9c75c1-ec86-4637-be65-48efdda20ba7-plugins-config-volume\") pod \"4f9c75c1-ec86-4637-be65-48efdda20ba7\" (UID: \"4f9c75c1-ec86-4637-be65-48efdda20ba7\") " Nov 03 09:08:19.738783 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.738730 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9c75c1-ec86-4637-be65-48efdda20ba7-kube-api-access-ql9st\" (UniqueName: \"kubernetes.io/projected/4f9c75c1-ec86-4637-be65-48efdda20ba7-kube-api-access-ql9st\") pod \"4f9c75c1-ec86-4637-be65-48efdda20ba7\" (UID: \"4f9c75c1-ec86-4637-be65-48efdda20ba7\") " Nov 03 09:08:19.738860 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.738836 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9c75c1-ec86-4637-be65-48efdda20ba7-plugins-config-volume" pod "4f9c75c1-ec86-4637-be65-48efdda20ba7" (UID: "4f9c75c1-ec86-4637-be65-48efdda20ba7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:19.739522 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.739439 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9c75c1-ec86-4637-be65-48efdda20ba7-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:19.747783 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.741459 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:08:19.747783 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.741536 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9c75c1-ec86-4637-be65-48efdda20ba7-kube-api-access-ql9st" pod "4f9c75c1-ec86-4637-be65-48efdda20ba7" (UID: "4f9c75c1-ec86-4637-be65-48efdda20ba7"). InnerVolumeSpecName "kube-api-access-ql9st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:19.840427 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.840345 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c50c64-e1d9-4f70-bf80-12713de54c41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c50c64-e1d9-4f70-bf80-12713de54c41-plugins-config-volume\") pod \"97c50c64-e1d9-4f70-bf80-12713de54c41\" (UID: \"97c50c64-e1d9-4f70-bf80-12713de54c41\") " Nov 03 09:08:19.840610 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.840595 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c50c64-e1d9-4f70-bf80-12713de54c41-kube-api-access-hlkbl\" (UniqueName: \"kubernetes.io/projected/97c50c64-e1d9-4f70-bf80-12713de54c41-kube-api-access-hlkbl\") pod \"97c50c64-e1d9-4f70-bf80-12713de54c41\" (UID: \"97c50c64-e1d9-4f70-bf80-12713de54c41\") " Nov 03 09:08:19.840844 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.840818 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c50c64-e1d9-4f70-bf80-12713de54c41-plugins-config-volume" pod "97c50c64-e1d9-4f70-bf80-12713de54c41" (UID: "97c50c64-e1d9-4f70-bf80-12713de54c41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:19.841212 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.841191 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql9st\" (UniqueName: \"kubernetes.io/projected/4f9c75c1-ec86-4637-be65-48efdda20ba7-kube-api-access-ql9st\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:19.841497 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.841217 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c50c64-e1d9-4f70-bf80-12713de54c41-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:19.842931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.842908 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c50c64-e1d9-4f70-bf80-12713de54c41-kube-api-access-hlkbl" pod "97c50c64-e1d9-4f70-bf80-12713de54c41" (UID: "97c50c64-e1d9-4f70-bf80-12713de54c41"). InnerVolumeSpecName "kube-api-access-hlkbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:19.942194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:19.942167 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlkbl\" (UniqueName: \"kubernetes.io/projected/97c50c64-e1d9-4f70-bf80-12713de54c41-kube-api-access-hlkbl\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:20.009161 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.009123 5015 generic.go:356] "Generic (PLEG): container finished" podID="4f9c75c1-ec86-4637-be65-48efdda20ba7" containerID="89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21" exitCode=0 Nov 03 09:08:20.009662 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.009186 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" Nov 03 09:08:20.009662 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.009208 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" event={"ID":"4f9c75c1-ec86-4637-be65-48efdda20ba7","Type":"ContainerDied","Data":"89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21"} Nov 03 09:08:20.009662 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.009264 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld" event={"ID":"4f9c75c1-ec86-4637-be65-48efdda20ba7","Type":"ContainerDied","Data":"20a1a9bdde59eec58613696a77a980b4302d5417080f50832b1aa7f448332361"} Nov 03 09:08:20.009662 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.009337 5015 scope.go:122] "RemoveContainer" containerID="89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21" Nov 03 09:08:20.011281 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.011189 5015 generic.go:356] "Generic (PLEG): container finished" podID="97c50c64-e1d9-4f70-bf80-12713de54c41" containerID="22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2" exitCode=0 Nov 03 09:08:20.011281 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.011234 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" Nov 03 09:08:20.011281 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.011251 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" event={"ID":"97c50c64-e1d9-4f70-bf80-12713de54c41","Type":"ContainerDied","Data":"22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2"} Nov 03 09:08:20.011541 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.011322 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb" event={"ID":"97c50c64-e1d9-4f70-bf80-12713de54c41","Type":"ContainerDied","Data":"ad2e62f887b0204de48d1df686aa85e1085b943569b17803beb6719baaee44cb"} Nov 03 09:08:20.012031 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.011988 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:20.013034 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.013016 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:08:20.023518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.023503 5015 scope.go:122] "RemoveContainer" containerID="89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21" Nov 03 09:08:20.023799 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:08:20.023782 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21\": container with ID starting with 89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21 not found: ID does not exist" containerID="89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21" Nov 03 09:08:20.023862 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.023805 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21"} err="failed to get container status \"89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21\": rpc error: code = NotFound desc = could not find container \"89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21\": container with ID starting with 89c4207d1fe2dfb824c7267fb2cc81f6f1681c427ffd10b23f7254b1f9b0cc21 not found: ID does not exist" Nov 03 09:08:20.023862 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.023820 5015 scope.go:122] "RemoveContainer" containerID="22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2" Nov 03 09:08:20.033704 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.033684 5015 scope.go:122] "RemoveContainer" containerID="22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2" Nov 03 09:08:20.033990 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:08:20.033972 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2\": container with ID starting with 22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2 not found: ID does not exist" containerID="22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2" Nov 03 09:08:20.034037 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.033998 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2"} err="failed to get container status \"22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2\": rpc error: code = NotFound desc = could not find container \"22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2\": container with ID starting with 22ec9ae6363b48f8e3ef802ac4b4d520521b10ac11d6df9f334aa613519ae6c2 not found: ID does not exist" Nov 03 09:08:20.180293 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.180217 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb"] Nov 03 09:08:20.185521 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.185495 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-204dfc-6d7cf599cf-shddb"] Nov 03 09:08:20.297682 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.297653 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld"] Nov 03 09:08:20.303104 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:20.303076 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-204dfc-74769d8d57-rpnld"] Nov 03 09:08:21.215463 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:21.215431 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9c75c1-ec86-4637-be65-48efdda20ba7" path="/var/lib/kubelet/pods/4f9c75c1-ec86-4637-be65-48efdda20ba7/volumes" Nov 03 09:08:21.215893 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:21.215877 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c50c64-e1d9-4f70-bf80-12713de54c41" path="/var/lib/kubelet/pods/97c50c64-e1d9-4f70-bf80-12713de54c41/volumes" Nov 03 09:08:25.718704 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.718660 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q"] Nov 03 09:08:25.719839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.719682 5015 cpu_manager.go:414] "containerMap: removing container" podUID="97c50c64-e1d9-4f70-bf80-12713de54c41" containerName="payload-pre-processing" Nov 03 09:08:25.719839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.719703 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c50c64-e1d9-4f70-bf80-12713de54c41" containerName="payload-pre-processing" Nov 03 09:08:25.719839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.719770 5015 cpu_manager.go:414] "containerMap: removing container" podUID="4f9c75c1-ec86-4637-be65-48efdda20ba7" containerName="payload-processing" Nov 03 09:08:25.719839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.719778 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9c75c1-ec86-4637-be65-48efdda20ba7" containerName="payload-processing" Nov 03 09:08:25.720123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.719959 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c50c64-e1d9-4f70-bf80-12713de54c41" containerName="payload-pre-processing" Nov 03 09:08:25.720123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.719983 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9c75c1-ec86-4637-be65-48efdda20ba7" containerName="payload-processing" Nov 03 09:08:25.724855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.724833 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:08:25.782705 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.782672 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q"] Nov 03 09:08:25.884517 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.833831 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4ae3db-dockercfg-56cj2\"" Nov 03 09:08:25.884517 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.833933 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4ae3db\"" Nov 03 09:08:25.977291 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.977247 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q\" (UID: \"6d22752b-b9f0-45ba-a8aa-1229378bd3ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:08:25.977686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:25.977656 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd5ts\" (UniqueName: \"kubernetes.io/projected/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-kube-api-access-pd5ts\") pod \"payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q\" (UID: \"6d22752b-b9f0-45ba-a8aa-1229378bd3ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:08:26.079685 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:26.079658 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd5ts\" (UniqueName: \"kubernetes.io/projected/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-kube-api-access-pd5ts\") pod \"payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q\" (UID: \"6d22752b-b9f0-45ba-a8aa-1229378bd3ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:08:26.079874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:26.079800 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q\" (UID: \"6d22752b-b9f0-45ba-a8aa-1229378bd3ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:08:26.084510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:26.081309 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q\" (UID: \"6d22752b-b9f0-45ba-a8aa-1229378bd3ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:08:26.143700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:26.143663 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd5ts\" (UniqueName: \"kubernetes.io/projected/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-kube-api-access-pd5ts\") pod \"payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q\" (UID: \"6d22752b-b9f0-45ba-a8aa-1229378bd3ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:08:26.340021 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:26.339941 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:08:26.731326 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:26.731294 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q"] Nov 03 09:08:26.732785 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:08:26.732742 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d22752b_b9f0_45ba_a8aa_1229378bd3ab.slice/crio-55219bb79dee8903b1385d986dafb37389b01f3a71b31a29fda0a020684d4d36 WatchSource:0}: Error finding container 55219bb79dee8903b1385d986dafb37389b01f3a71b31a29fda0a020684d4d36: Status 404 returned error can't find the container with id 55219bb79dee8903b1385d986dafb37389b01f3a71b31a29fda0a020684d4d36 Nov 03 09:08:27.087581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:27.085027 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" event={"ID":"6d22752b-b9f0-45ba-a8aa-1229378bd3ab","Type":"ContainerStarted","Data":"aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47"} Nov 03 09:08:27.087581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:27.085101 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" event={"ID":"6d22752b-b9f0-45ba-a8aa-1229378bd3ab","Type":"ContainerStarted","Data":"55219bb79dee8903b1385d986dafb37389b01f3a71b31a29fda0a020684d4d36"} Nov 03 09:08:27.087581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:27.086275 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:08:27.207463 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:27.207404 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" podStartSLOduration=2.207385765 podStartE2EDuration="2.207385765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:08:27.207140768 +0000 UTC m=+1740.132082986" watchObservedRunningTime="2026-11-03 09:08:27.207385765 +0000 UTC m=+1740.132327982" Nov 03 09:08:27.706460 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:27.706428 5015 scope.go:122] "RemoveContainer" containerID="def7213e5ccb260d00f8ea660e675a3ff1edff1541fbfc9ff8d0c0e453a59486" Nov 03 09:08:27.717742 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:27.717719 5015 scope.go:122] "RemoveContainer" containerID="299bdfbf871469deadcc3c30d938436ea745ebfb19955d5356d48697d6987cfc" Nov 03 09:08:27.727814 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:27.727784 5015 scope.go:122] "RemoveContainer" containerID="83c0df5071456ad501f7e68c381bcd736e39c1fc0b6f3e5ca392c8bddde0f20f" Nov 03 09:08:27.738177 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:27.738147 5015 scope.go:122] "RemoveContainer" containerID="c0af727515840d8d6931586fc62ec416bc2fc7b156b213d8994c8d381cb96703" Nov 03 09:08:30.328505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:30.328445 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c"] Nov 03 09:08:30.328930 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:30.328656 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerName="istio-proxy" containerID="cri-o://d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3" gracePeriod=30 Nov 03 09:08:30.379242 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:30.379208 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g"] Nov 03 09:08:30.379461 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:30.379408 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" containerName="istio-proxy" containerID="cri-o://5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a" gracePeriod=30 Nov 03 09:08:30.383784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:30.383759 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8"] Nov 03 09:08:30.383964 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:30.383930 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerName="istio-proxy" containerID="cri-o://a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a" gracePeriod=30 Nov 03 09:08:30.430200 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:30.430163 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f"] Nov 03 09:08:30.430417 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:30.430394 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" containerID="cri-o://c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b" gracePeriod=30 Nov 03 09:08:32.476385 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:32.476342 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:08:32.476905 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:32.476426 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:08:35.178345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:35.178252 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:08:35.178345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:35.178322 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:08:35.631760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:35.607854 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg"] Nov 03 09:08:35.631760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:35.608185 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" podUID="90342bf6-bae8-4bc6-8fe4-c538d0ac63f6" containerName="maas-api" containerID="cri-o://5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457" gracePeriod=30 Nov 03 09:08:35.672417 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:35.669005 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 09:08:35.672417 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:35.669074 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 09:08:35.895248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:35.894901 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:08:35.985347 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:35.985256 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:08:36.001998 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.000699 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:08:36.038123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.037807 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-token\" (UniqueName: \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-token\") pod \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " Nov 03 09:08:36.038791 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.038765 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-credential-socket\") pod \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " Nov 03 09:08:36.038898 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.038843 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-certs\") pod \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " Nov 03 09:08:36.038961 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.038923 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-socket\") pod \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " Nov 03 09:08:36.039019 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.038963 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-kube-api-access-wd5pj\" (UniqueName: \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-kube-api-access-wd5pj\") pod \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " Nov 03 09:08:36.039764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.039085 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-credential-socket" pod "1ffd055e-1668-4ca2-8495-6fe0a3cc5923" (UID: "1ffd055e-1668-4ca2-8495-6fe0a3cc5923"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.039764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.039103 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istiod-ca-cert\") pod \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " Nov 03 09:08:36.039764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.039175 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-data\") pod \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " Nov 03 09:08:36.039764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.039230 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-podinfo\") pod \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " Nov 03 09:08:36.039764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.039387 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-envoy\") pod \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\" (UID: \"1ffd055e-1668-4ca2-8495-6fe0a3cc5923\") " Nov 03 09:08:36.039764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.039450 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-data" pod "1ffd055e-1668-4ca2-8495-6fe0a3cc5923" (UID: "1ffd055e-1668-4ca2-8495-6fe0a3cc5923"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.040176 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.039946 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istiod-ca-cert" pod "1ffd055e-1668-4ca2-8495-6fe0a3cc5923" (UID: "1ffd055e-1668-4ca2-8495-6fe0a3cc5923"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:36.040239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.040179 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-socket" pod "1ffd055e-1668-4ca2-8495-6fe0a3cc5923" (UID: "1ffd055e-1668-4ca2-8495-6fe0a3cc5923"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.040239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.040171 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.040239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.040210 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.040239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.040221 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.040809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.040757 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-certs" pod "1ffd055e-1668-4ca2-8495-6fe0a3cc5923" (UID: "1ffd055e-1668-4ca2-8495-6fe0a3cc5923"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.042596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.042573 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-token" pod "1ffd055e-1668-4ca2-8495-6fe0a3cc5923" (UID: "1ffd055e-1668-4ca2-8495-6fe0a3cc5923"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:36.042975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.042945 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-envoy" pod "1ffd055e-1668-4ca2-8495-6fe0a3cc5923" (UID: "1ffd055e-1668-4ca2-8495-6fe0a3cc5923"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.043927 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.043901 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-kube-api-access-wd5pj" pod "1ffd055e-1668-4ca2-8495-6fe0a3cc5923" (UID: "1ffd055e-1668-4ca2-8495-6fe0a3cc5923"). InnerVolumeSpecName "kube-api-access-wd5pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:36.045606 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.045465 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-podinfo" pod "1ffd055e-1668-4ca2-8495-6fe0a3cc5923" (UID: "1ffd055e-1668-4ca2-8495-6fe0a3cc5923"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:08:36.056311 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.056289 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 09:08:36.107733 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.107708 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:08:36.141371 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.141326 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-envoy\") pod \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " Nov 03 09:08:36.141689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.141373 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bc7a494-d07b-4693-bc27-31359cb34014-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc7a494-d07b-4693-bc27-31359cb34014-istio-podinfo\") pod \"1bc7a494-d07b-4693-bc27-31359cb34014\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " Nov 03 09:08:36.141689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.141405 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-credential-socket\") pod \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " Nov 03 09:08:36.141689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.141451 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-podinfo\") pod \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " Nov 03 09:08:36.141689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.141503 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-kube-api-access-zv7n2\" (UniqueName: \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-kube-api-access-zv7n2\") pod \"1bc7a494-d07b-4693-bc27-31359cb34014\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " Nov 03 09:08:36.141689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.141539 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istiod-ca-cert\") pod \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " Nov 03 09:08:36.141689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.141564 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-token\" (UniqueName: \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-token\") pod \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " Nov 03 09:08:36.141689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.141640 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-kube-api-access-b7kgr\" (UniqueName: \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-kube-api-access-b7kgr\") pod \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " Nov 03 09:08:36.141689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.141666 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-socket\") pod \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " Nov 03 09:08:36.141689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.141692 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-envoy\") pod \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " Nov 03 09:08:36.142222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.141731 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-credential-socket\") pod \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " Nov 03 09:08:36.142222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.142027 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-socket" pod "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" (UID: "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.142222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.142050 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-credential-socket" pod "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" (UID: "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.142371 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.142306 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-credential-socket" pod "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" (UID: "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.142635 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-envoy\") pod \"1bc7a494-d07b-4693-bc27-31359cb34014\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.142711 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-istio-token\" (UniqueName: \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-istio-token\") pod \"1bc7a494-d07b-4693-bc27-31359cb34014\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.142785 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-socket\") pod \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.142820 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-credential-socket\") pod \"1bc7a494-d07b-4693-bc27-31359cb34014\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.142875 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-data\") pod \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.142942 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-kube-api-access-ggln6\" (UniqueName: \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-kube-api-access-ggln6\") pod \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.143020 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-certs\") pod \"1bc7a494-d07b-4693-bc27-31359cb34014\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.143082 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-socket\") pod \"1bc7a494-d07b-4693-bc27-31359cb34014\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.143186 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-podinfo\") pod \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.143274 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-certs\") pod \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.143306 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-data\") pod \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.143346 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istiod-ca-cert\") pod \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.143384 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc7a494-d07b-4693-bc27-31359cb34014-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc7a494-d07b-4693-bc27-31359cb34014-istiod-ca-cert\") pod \"1bc7a494-d07b-4693-bc27-31359cb34014\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.143546 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-certs\") pod \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\" (UID: \"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.143596 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-token\" (UniqueName: \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-token\") pod \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\" (UID: \"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4\") " Nov 03 09:08:36.144425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.143780 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-data\") pod \"1bc7a494-d07b-4693-bc27-31359cb34014\" (UID: \"1bc7a494-d07b-4693-bc27-31359cb34014\") " Nov 03 09:08:36.145952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.145569 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-envoy" pod "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" (UID: "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.145952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.145663 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.145952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.145681 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.145952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.145691 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.145952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.145701 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.145952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.145709 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.145952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.145718 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.145952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.145725 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.145952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.145736 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd5pj\" (UniqueName: \"kubernetes.io/projected/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-kube-api-access-wd5pj\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.145952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.145744 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffd055e-1668-4ca2-8495-6fe0a3cc5923-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.145952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.145774 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-socket" pod "1bc7a494-d07b-4693-bc27-31359cb34014" (UID: "1bc7a494-d07b-4693-bc27-31359cb34014"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.147430 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.146655 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-certs" pod "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" (UID: "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.147430 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.147077 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istiod-ca-cert" pod "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" (UID: "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:36.147628 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.147601 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-certs" pod "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" (UID: "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.150490 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.148230 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-socket" pod "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" (UID: "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.150490 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.148628 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-credential-socket" pod "1bc7a494-d07b-4693-bc27-31359cb34014" (UID: "1bc7a494-d07b-4693-bc27-31359cb34014"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.150490 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.148832 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-data" pod "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" (UID: "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.150490 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.148934 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-data" pod "1bc7a494-d07b-4693-bc27-31359cb34014" (UID: "1bc7a494-d07b-4693-bc27-31359cb34014"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.150490 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.149010 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-podinfo" pod "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" (UID: "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:08:36.150490 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.149901 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istiod-ca-cert" pod "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" (UID: "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:36.150490 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.150175 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-token" pod "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" (UID: "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:36.150490 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.150226 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc7a494-d07b-4693-bc27-31359cb34014-istiod-ca-cert" pod "1bc7a494-d07b-4693-bc27-31359cb34014" (UID: "1bc7a494-d07b-4693-bc27-31359cb34014"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:36.150490 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.150448 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-certs" pod "1bc7a494-d07b-4693-bc27-31359cb34014" (UID: "1bc7a494-d07b-4693-bc27-31359cb34014"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.151026 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.150613 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-data" pod "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" (UID: "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.151026 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.150731 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-istio-token" pod "1bc7a494-d07b-4693-bc27-31359cb34014" (UID: "1bc7a494-d07b-4693-bc27-31359cb34014"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:36.152934 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.152911 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-envoy" pod "1bc7a494-d07b-4693-bc27-31359cb34014" (UID: "1bc7a494-d07b-4693-bc27-31359cb34014"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.152934 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.152919 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-token" pod "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" (UID: "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:36.152934 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.152908 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-envoy" pod "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" (UID: "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:36.153154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.152991 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-kube-api-access-zv7n2" pod "1bc7a494-d07b-4693-bc27-31359cb34014" (UID: "1bc7a494-d07b-4693-bc27-31359cb34014"). InnerVolumeSpecName "kube-api-access-zv7n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:36.153778 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.153307 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-kube-api-access-b7kgr" pod "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" (UID: "762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4"). InnerVolumeSpecName "kube-api-access-b7kgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:36.154261 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.154238 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-kube-api-access-ggln6" pod "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" (UID: "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d"). InnerVolumeSpecName "kube-api-access-ggln6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:36.154582 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.154559 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bc7a494-d07b-4693-bc27-31359cb34014-istio-podinfo" pod "1bc7a494-d07b-4693-bc27-31359cb34014" (UID: "1bc7a494-d07b-4693-bc27-31359cb34014"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:08:36.154706 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.154688 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-podinfo" pod "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" (UID: "bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:08:36.163766 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.163735 5015 generic.go:356] "Generic (PLEG): container finished" podID="90342bf6-bae8-4bc6-8fe4-c538d0ac63f6" containerID="5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457" exitCode=0 Nov 03 09:08:36.163885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.163841 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" Nov 03 09:08:36.163885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.163862 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" event={"ID":"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6","Type":"ContainerDied","Data":"5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457"} Nov 03 09:08:36.164003 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.163916 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg" event={"ID":"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6","Type":"ContainerDied","Data":"799c281a44c11da1f2bbdd8b925ed75c60e67f89a180dcdc65617a31ad9aa7d3"} Nov 03 09:08:36.164003 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.163938 5015 scope.go:122] "RemoveContainer" containerID="5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457" Nov 03 09:08:36.165957 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.165933 5015 generic.go:356] "Generic (PLEG): container finished" podID="1bc7a494-d07b-4693-bc27-31359cb34014" containerID="5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a" exitCode=0 Nov 03 09:08:36.166065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.166047 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" event={"ID":"1bc7a494-d07b-4693-bc27-31359cb34014","Type":"ContainerDied","Data":"5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a"} Nov 03 09:08:36.166135 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.166096 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" event={"ID":"1bc7a494-d07b-4693-bc27-31359cb34014","Type":"ContainerDied","Data":"5c77d0aad7bafd2860174fe6461bfdcaebb3e5922edd58c0cc1a45c52d5785d2"} Nov 03 09:08:36.166191 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.166167 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g" Nov 03 09:08:36.178304 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.178285 5015 scope.go:122] "RemoveContainer" containerID="5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457" Nov 03 09:08:36.178370 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.178307 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" event={"ID":"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d","Type":"ContainerDied","Data":"a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a"} Nov 03 09:08:36.178370 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.178345 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" Nov 03 09:08:36.178812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.178285 5015 generic.go:356] "Generic (PLEG): container finished" podID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerID="a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a" exitCode=0 Nov 03 09:08:36.178812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.178605 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8" event={"ID":"bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d","Type":"ContainerDied","Data":"f6ad44e74cebb1d5198f331fd5154cbf70ae4fb1129225982c2ea615541f37d0"} Nov 03 09:08:36.178812 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:08:36.178644 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457\": container with ID starting with 5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457 not found: ID does not exist" containerID="5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457" Nov 03 09:08:36.178812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.178680 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457"} err="failed to get container status \"5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457\": rpc error: code = NotFound desc = could not find container \"5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457\": container with ID starting with 5c4487ae6538b3f754ddf90b16f1a51f80c77c843840a95b9ea78561100df457 not found: ID does not exist" Nov 03 09:08:36.178812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.178708 5015 scope.go:122] "RemoveContainer" containerID="5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a" Nov 03 09:08:36.180621 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.180597 5015 generic.go:356] "Generic (PLEG): container finished" podID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerID="d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3" exitCode=0 Nov 03 09:08:36.180721 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.180638 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" Nov 03 09:08:36.180721 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.180684 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" event={"ID":"1ffd055e-1668-4ca2-8495-6fe0a3cc5923","Type":"ContainerDied","Data":"d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3"} Nov 03 09:08:36.180823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.180737 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" event={"ID":"1ffd055e-1668-4ca2-8495-6fe0a3cc5923","Type":"ContainerDied","Data":"8fb2978ad088b1d026d02f9464349b9e04caeda72c8024c8c469b4bb0ff26667"} Nov 03 09:08:36.182362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.182340 5015 generic.go:356] "Generic (PLEG): container finished" podID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerID="c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b" exitCode=0 Nov 03 09:08:36.182451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.182402 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" event={"ID":"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4","Type":"ContainerDied","Data":"c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b"} Nov 03 09:08:36.182451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.182427 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" event={"ID":"762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4","Type":"ContainerDied","Data":"7025033a3839505a4e0ee39c3f38fc640a3097ad02eceec1af6e65a5dfab82a8"} Nov 03 09:08:36.182589 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.182497 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f" Nov 03 09:08:36.206467 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.206442 5015 scope.go:122] "RemoveContainer" containerID="5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a" Nov 03 09:08:36.206882 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:08:36.206859 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a\": container with ID starting with 5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a not found: ID does not exist" containerID="5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a" Nov 03 09:08:36.206968 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.206890 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a"} err="failed to get container status \"5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a\": rpc error: code = NotFound desc = could not find container \"5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a\": container with ID starting with 5a90b37e656feb6f047fc4f40336b4e10151f730be05b6c8b78ccf4bffd7a31a not found: ID does not exist" Nov 03 09:08:36.206968 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.206910 5015 scope.go:122] "RemoveContainer" containerID="a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a" Nov 03 09:08:36.217048 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.217027 5015 scope.go:122] "RemoveContainer" containerID="a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a" Nov 03 09:08:36.217338 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:08:36.217315 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a\": container with ID starting with a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a not found: ID does not exist" containerID="a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a" Nov 03 09:08:36.217403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.217346 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a"} err="failed to get container status \"a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a\": rpc error: code = NotFound desc = could not find container \"a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a\": container with ID starting with a85b6b21be8ba48559f595696a75355075f1121776ff87b7d5c9e03c9242b05a not found: ID does not exist" Nov 03 09:08:36.217403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.217364 5015 scope.go:122] "RemoveContainer" containerID="d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3" Nov 03 09:08:36.226888 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.226870 5015 scope.go:122] "RemoveContainer" containerID="d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3" Nov 03 09:08:36.227163 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:08:36.227140 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3\": container with ID starting with d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3 not found: ID does not exist" containerID="d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3" Nov 03 09:08:36.227211 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.227171 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3"} err="failed to get container status \"d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3\": rpc error: code = NotFound desc = could not find container \"d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3\": container with ID starting with d453f7cf16caf9eae71dd977f716bab29a65d85a6fcee00beeb9794e5f4f1aa3 not found: ID does not exist" Nov 03 09:08:36.227211 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.227188 5015 scope.go:122] "RemoveContainer" containerID="c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b" Nov 03 09:08:36.238081 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.238060 5015 scope.go:122] "RemoveContainer" containerID="c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b" Nov 03 09:08:36.238447 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:08:36.238382 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b\": container with ID starting with c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b not found: ID does not exist" containerID="c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b" Nov 03 09:08:36.238447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.238417 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b"} err="failed to get container status \"c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b\": rpc error: code = NotFound desc = could not find container \"c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b\": container with ID starting with c9a7597ad3559c958e05cfcf03788775f04175044771adcbd6d61728ef69776b not found: ID does not exist" Nov 03 09:08:36.246625 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.246601 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-maas-api-tls\") pod \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " Nov 03 09:08:36.246748 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.246641 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-metrics-certs\") pod \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " Nov 03 09:08:36.246816 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.246790 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-openshift-service-ca\") pod \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " Nov 03 09:08:36.246915 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.246902 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-kube-api-access-gr4z5\" (UniqueName: \"kubernetes.io/projected/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-kube-api-access-gr4z5\") pod \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\" (UID: \"90342bf6-bae8-4bc6-8fe4-c538d0ac63f6\") " Nov 03 09:08:36.247493 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247436 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-openshift-service-ca" pod "90342bf6-bae8-4bc6-8fe4-c538d0ac63f6" (UID: "90342bf6-bae8-4bc6-8fe4-c538d0ac63f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:36.247810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247701 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.247810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247730 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.247810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247745 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.247810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247761 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.247810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247778 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.247810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247792 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggln6\" (UniqueName: \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-kube-api-access-ggln6\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.247810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247808 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247822 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247836 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247851 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247867 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247881 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247897 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc7a494-d07b-4693-bc27-31359cb34014-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247912 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247927 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247940 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247955 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc7a494-d07b-4693-bc27-31359cb34014-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247968 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247985 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc7a494-d07b-4693-bc27-31359cb34014-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.247998 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.248014 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv7n2\" (UniqueName: \"kubernetes.io/projected/1bc7a494-d07b-4693-bc27-31359cb34014-kube-api-access-zv7n2\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.248029 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.248043 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.248058 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7kgr\" (UniqueName: \"kubernetes.io/projected/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4-kube-api-access-b7kgr\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.248317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.248071 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.249370 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.249109 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-metrics-certs" pod "90342bf6-bae8-4bc6-8fe4-c538d0ac63f6" (UID: "90342bf6-bae8-4bc6-8fe4-c538d0ac63f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:08:36.249427 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.249372 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-maas-api-tls" pod "90342bf6-bae8-4bc6-8fe4-c538d0ac63f6" (UID: "90342bf6-bae8-4bc6-8fe4-c538d0ac63f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:08:36.249639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.249622 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-kube-api-access-gr4z5" pod "90342bf6-bae8-4bc6-8fe4-c538d0ac63f6" (UID: "90342bf6-bae8-4bc6-8fe4-c538d0ac63f6"). InnerVolumeSpecName "kube-api-access-gr4z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:36.304437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.304339 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7"] Nov 03 09:08:36.305415 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305383 5015 cpu_manager.go:414] "containerMap: removing container" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerName="istio-proxy" Nov 03 09:08:36.305415 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305414 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerName="istio-proxy" Nov 03 09:08:36.305644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305463 5015 cpu_manager.go:414] "containerMap: removing container" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" Nov 03 09:08:36.305644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305507 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" Nov 03 09:08:36.305644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305527 5015 cpu_manager.go:414] "containerMap: removing container" podUID="90342bf6-bae8-4bc6-8fe4-c538d0ac63f6" containerName="maas-api" Nov 03 09:08:36.305644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305535 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90342bf6-bae8-4bc6-8fe4-c538d0ac63f6" containerName="maas-api" Nov 03 09:08:36.305644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305557 5015 cpu_manager.go:414] "containerMap: removing container" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" containerName="istio-proxy" Nov 03 09:08:36.305644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305564 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" containerName="istio-proxy" Nov 03 09:08:36.305644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305598 5015 cpu_manager.go:414] "containerMap: removing container" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerName="istio-proxy" Nov 03 09:08:36.305644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305605 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerName="istio-proxy" Nov 03 09:08:36.305880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305746 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" containerName="istio-proxy" Nov 03 09:08:36.305880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305755 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="90342bf6-bae8-4bc6-8fe4-c538d0ac63f6" containerName="maas-api" Nov 03 09:08:36.305880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305763 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" containerName="istio-proxy" Nov 03 09:08:36.305880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305772 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" containerName="istio-proxy" Nov 03 09:08:36.305880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.305780 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerName="istio-proxy" Nov 03 09:08:36.310978 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.310960 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:08:36.349092 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.349064 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.349092 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.349094 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr4z5\" (UniqueName: \"kubernetes.io/projected/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-kube-api-access-gr4z5\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.349374 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.349108 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:36.365614 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.365589 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b0c653f4\"" Nov 03 09:08:36.365745 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.365640 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b0c653f4-dockercfg-pwvzh\"" Nov 03 09:08:36.367011 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.366988 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7"] Nov 03 09:08:36.474502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.467713 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cf2c34-2317-4277-bb48-b76e62e75fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7\" (UID: \"49cf2c34-2317-4277-bb48-b76e62e75fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:08:36.474502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.467767 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlrcb\" (UniqueName: \"kubernetes.io/projected/49cf2c34-2317-4277-bb48-b76e62e75fe1-kube-api-access-vlrcb\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7\" (UID: \"49cf2c34-2317-4277-bb48-b76e62e75fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:08:36.493499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.486348 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f"] Nov 03 09:08:36.533706 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.516220 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cqr9f"] Nov 03 09:08:36.563232 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.562928 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg"] Nov 03 09:08:36.569658 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.569625 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cf2c34-2317-4277-bb48-b76e62e75fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7\" (UID: \"49cf2c34-2317-4277-bb48-b76e62e75fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:08:36.571404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.571375 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlrcb\" (UniqueName: \"kubernetes.io/projected/49cf2c34-2317-4277-bb48-b76e62e75fe1-kube-api-access-vlrcb\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7\" (UID: \"49cf2c34-2317-4277-bb48-b76e62e75fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:08:36.572118 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.572077 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cf2c34-2317-4277-bb48-b76e62e75fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7\" (UID: \"49cf2c34-2317-4277-bb48-b76e62e75fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:08:36.574050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.572377 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:08:36.575921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.575891 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg"] Nov 03 09:08:36.638367 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.638334 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g"] Nov 03 09:08:36.646030 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.645994 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6k4g"] Nov 03 09:08:36.647074 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.647047 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlrcb\" (UniqueName: \"kubernetes.io/projected/49cf2c34-2317-4277-bb48-b76e62e75fe1-kube-api-access-vlrcb\") pod \"payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7\" (UID: \"49cf2c34-2317-4277-bb48-b76e62e75fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:08:36.673904 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.673852 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485174e2-1192-4b2f-abaf-f42b74c6a557-plugins-config-volume\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg\" (UID: \"485174e2-1192-4b2f-abaf-f42b74c6a557\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:08:36.674063 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.673950 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqx8l\" (UniqueName: \"kubernetes.io/projected/485174e2-1192-4b2f-abaf-f42b74c6a557-kube-api-access-vqx8l\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg\" (UID: \"485174e2-1192-4b2f-abaf-f42b74c6a557\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:08:36.721121 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.721079 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:08:36.721337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.721158 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:08:36.776240 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.776190 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485174e2-1192-4b2f-abaf-f42b74c6a557-plugins-config-volume\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg\" (UID: \"485174e2-1192-4b2f-abaf-f42b74c6a557\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:08:36.776676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.776648 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqx8l\" (UniqueName: \"kubernetes.io/projected/485174e2-1192-4b2f-abaf-f42b74c6a557-kube-api-access-vqx8l\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg\" (UID: \"485174e2-1192-4b2f-abaf-f42b74c6a557\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:08:36.778230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.778206 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485174e2-1192-4b2f-abaf-f42b74c6a557-plugins-config-volume\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg\" (UID: \"485174e2-1192-4b2f-abaf-f42b74c6a557\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:08:36.844409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.844213 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqx8l\" (UniqueName: \"kubernetes.io/projected/485174e2-1192-4b2f-abaf-f42b74c6a557-kube-api-access-vqx8l\") pod \"payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg\" (UID: \"485174e2-1192-4b2f-abaf-f42b74c6a557\") " pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:08:36.860836 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.857592 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8"] Nov 03 09:08:36.869132 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.867523 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-bsgn8"] Nov 03 09:08:36.901359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.901319 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:08:36.926500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.926380 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:08:36.970174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.970146 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c"] Nov 03 09:08:36.976816 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:36.976785 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-9m66c"] Nov 03 09:08:37.207739 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:37.207663 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc7a494-d07b-4693-bc27-31359cb34014" path="/var/lib/kubelet/pods/1bc7a494-d07b-4693-bc27-31359cb34014/volumes" Nov 03 09:08:37.208173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:37.208159 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffd055e-1668-4ca2-8495-6fe0a3cc5923" path="/var/lib/kubelet/pods/1ffd055e-1668-4ca2-8495-6fe0a3cc5923/volumes" Nov 03 09:08:37.208642 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:37.208628 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4" path="/var/lib/kubelet/pods/762691b8-4d8b-41b4-b0fd-b1f64b7f6fe4/volumes" Nov 03 09:08:37.209404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:37.209388 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d" path="/var/lib/kubelet/pods/bb7cc2ee-3c0b-48fd-9701-87b4c692aa9d/volumes" Nov 03 09:08:37.301625 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:37.301598 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg"] Nov 03 09:08:37.301766 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:08:37.301727 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485174e2_1192_4b2f_abaf_f42b74c6a557.slice/crio-029e1cc9fa3012a97d3e9a8af553687ccad191a0d25330e8a93e243cf5a9d740 WatchSource:0}: Error finding container 029e1cc9fa3012a97d3e9a8af553687ccad191a0d25330e8a93e243cf5a9d740: Status 404 returned error can't find the container with id 029e1cc9fa3012a97d3e9a8af553687ccad191a0d25330e8a93e243cf5a9d740 Nov 03 09:08:37.317383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:37.317329 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg"] Nov 03 09:08:37.322857 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:37.322836 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-204dfc-67cb8f5df5-5g4hg"] Nov 03 09:08:37.356015 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:37.355983 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7"] Nov 03 09:08:37.617502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:37.617420 5015 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal" podUID="6652f123a319d675474cd7afd732c542" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:08:38.204208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:38.204174 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" event={"ID":"49cf2c34-2317-4277-bb48-b76e62e75fe1","Type":"ContainerStarted","Data":"54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341"} Nov 03 09:08:38.204395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:38.204238 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" event={"ID":"49cf2c34-2317-4277-bb48-b76e62e75fe1","Type":"ContainerStarted","Data":"8d77716c0bf35b7386d4d8077401209d5920857f69580f295d739daa355bc0e5"} Nov 03 09:08:38.204395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:38.204268 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:08:38.206020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:38.205993 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" event={"ID":"485174e2-1192-4b2f-abaf-f42b74c6a557","Type":"ContainerStarted","Data":"f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706"} Nov 03 09:08:38.206145 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:38.206039 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" event={"ID":"485174e2-1192-4b2f-abaf-f42b74c6a557","Type":"ContainerStarted","Data":"029e1cc9fa3012a97d3e9a8af553687ccad191a0d25330e8a93e243cf5a9d740"} Nov 03 09:08:38.206145 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:38.206079 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:08:38.320755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:38.320701 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" podStartSLOduration=2.320685864 podStartE2EDuration="2.320685864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:08:38.319323361 +0000 UTC m=+1751.244265602" watchObservedRunningTime="2026-11-03 09:08:38.320685864 +0000 UTC m=+1751.245628080" Nov 03 09:08:38.438697 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:38.438608 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" podStartSLOduration=2.438590461 podStartE2EDuration="2.438590461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:08:38.434949493 +0000 UTC m=+1751.359891713" watchObservedRunningTime="2026-11-03 09:08:38.438590461 +0000 UTC m=+1751.363532677" Nov 03 09:08:39.107535 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:39.107506 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:08:39.207211 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:39.207182 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90342bf6-bae8-4bc6-8fe4-c538d0ac63f6" path="/var/lib/kubelet/pods/90342bf6-bae8-4bc6-8fe4-c538d0ac63f6/volumes" Nov 03 09:08:41.745274 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.745239 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7"] Nov 03 09:08:41.750907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.750876 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.810184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.810150 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7"] Nov 03 09:08:41.851256 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.851220 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-data\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.851553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.851271 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-podinfo\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.851553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.851369 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-envoy\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.851553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.851527 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-certs\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.851752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.851649 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c57ds\" (UniqueName: \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-kube-api-access-c57ds\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.851849 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.851829 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-socket\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.851974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.851958 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665332b4-9c0b-4a43-ab38-2050abc0521c-istiod-ca-cert\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.852034 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.852005 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-credential-socket\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.852100 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.852066 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-token\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.854533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.854507 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c32d41-openshift-default-dockercfg-st4km\"" Nov 03 09:08:41.867591 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.867563 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8"] Nov 03 09:08:41.952797 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.952756 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-envoy\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.952970 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.952815 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-certs\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.952970 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.952887 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c57ds\" (UniqueName: \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-kube-api-access-c57ds\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.953097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.952980 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-socket\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.953159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.953102 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665332b4-9c0b-4a43-ab38-2050abc0521c-istiod-ca-cert\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.953159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.953144 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-credential-socket\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.953262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.953208 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-token\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.953262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.953235 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-data\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.953359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.953265 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-podinfo\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.953359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.953343 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-certs\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.953510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.953465 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-socket\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.953677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.953657 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-data\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.954110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.954084 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665332b4-9c0b-4a43-ab38-2050abc0521c-istiod-ca-cert\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.954373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.954334 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-credential-socket\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.955605 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.955579 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-envoy\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:41.956364 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:41.956343 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-podinfo\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:42.015709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:42.015643 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-token\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:42.015839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:42.015753 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c57ds\" (UniqueName: \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-kube-api-access-c57ds\") pod \"e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:42.067342 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:42.067308 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:42.580562 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:42.578306 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:08:42.580562 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:42.578412 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:08:42.580562 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:42.578455 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:08:42.580869 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:42.580699 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7"] Nov 03 09:08:43.250940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:43.250897 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" event={"ID":"665332b4-9c0b-4a43-ab38-2050abc0521c","Type":"ContainerStarted","Data":"8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3"} Nov 03 09:08:43.251524 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:43.250965 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" event={"ID":"665332b4-9c0b-4a43-ab38-2050abc0521c","Type":"ContainerStarted","Data":"86f71f5ec9ffbc0ca25ed922871248aafb0413afaabd18aee55131d356443251"} Nov 03 09:08:43.374001 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:43.373928 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" podStartSLOduration=2.373913483 podStartE2EDuration="2.373913483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:08:43.372668026 +0000 UTC m=+1756.297610244" watchObservedRunningTime="2026-11-03 09:08:43.373913483 +0000 UTC m=+1756.298855743" Nov 03 09:08:44.067529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:44.067459 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:44.074275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:44.074249 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:44.261616 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:44.261582 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:44.262620 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:44.262596 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:08:46.334931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.334897 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6"] Nov 03 09:08:46.342807 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.342777 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:08:46.347699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.347669 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c297a297-b068-459e-a32c-c56bdb9b317f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6\" (UID: \"c297a297-b068-459e-a32c-c56bdb9b317f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:08:46.347919 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.347900 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7rhd\" (UniqueName: \"kubernetes.io/projected/c297a297-b068-459e-a32c-c56bdb9b317f-kube-api-access-x7rhd\") pod \"payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6\" (UID: \"c297a297-b068-459e-a32c-c56bdb9b317f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:08:46.398211 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.398178 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6"] Nov 03 09:08:46.452580 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.444769 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bcc5ce\"" Nov 03 09:08:46.452580 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.444928 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bcc5ce-dockercfg-z4zgq\"" Nov 03 09:08:46.452580 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.450401 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c297a297-b068-459e-a32c-c56bdb9b317f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6\" (UID: \"c297a297-b068-459e-a32c-c56bdb9b317f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:08:46.452580 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.450506 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7rhd\" (UniqueName: \"kubernetes.io/projected/c297a297-b068-459e-a32c-c56bdb9b317f-kube-api-access-x7rhd\") pod \"payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6\" (UID: \"c297a297-b068-459e-a32c-c56bdb9b317f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:08:46.454357 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.454327 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c297a297-b068-459e-a32c-c56bdb9b317f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6\" (UID: \"c297a297-b068-459e-a32c-c56bdb9b317f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:08:46.559507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.555567 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7rhd\" (UniqueName: \"kubernetes.io/projected/c297a297-b068-459e-a32c-c56bdb9b317f-kube-api-access-x7rhd\") pod \"payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6\" (UID: \"c297a297-b068-459e-a32c-c56bdb9b317f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:08:46.568968 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.564515 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8"] Nov 03 09:08:46.629751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.627694 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8"] Nov 03 09:08:46.642906 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.642879 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:08:46.666716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.662810 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:08:46.686519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.680776 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8x9s\" (UniqueName: \"kubernetes.io/projected/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-kube-api-access-b8x9s\") pod \"payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8\" (UID: \"9a24a04f-8d4f-4cbe-b131-78aa27d593c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:08:46.686519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.680828 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8\" (UID: \"9a24a04f-8d4f-4cbe-b131-78aa27d593c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:08:46.783592 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.783124 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x9s\" (UniqueName: \"kubernetes.io/projected/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-kube-api-access-b8x9s\") pod \"payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8\" (UID: \"9a24a04f-8d4f-4cbe-b131-78aa27d593c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:08:46.784944 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.784577 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8\" (UID: \"9a24a04f-8d4f-4cbe-b131-78aa27d593c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:08:46.785252 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.785203 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8\" (UID: \"9a24a04f-8d4f-4cbe-b131-78aa27d593c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:08:46.847533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.847491 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8x9s\" (UniqueName: \"kubernetes.io/projected/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-kube-api-access-b8x9s\") pod \"payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8\" (UID: \"9a24a04f-8d4f-4cbe-b131-78aa27d593c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:08:46.868605 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.868556 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" podUID="6e38970c-7160-4875-86b9-90b48d1a0c9e" containerName="payload-processing" containerID="cri-o://b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de" gracePeriod=25 Nov 03 09:08:46.908980 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:08:46.908947 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:08:46.994275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:46.994240 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:08:47.106245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.105317 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6"] Nov 03 09:08:47.316067 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.315504 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:08:47.377462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.377434 5015 generic.go:356] "Generic (PLEG): container finished" podID="6e38970c-7160-4875-86b9-90b48d1a0c9e" containerID="b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de" exitCode=0 Nov 03 09:08:47.377953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.377542 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" Nov 03 09:08:47.377953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.377629 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" event={"ID":"6e38970c-7160-4875-86b9-90b48d1a0c9e","Type":"ContainerDied","Data":"b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de"} Nov 03 09:08:47.377953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.377705 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8" event={"ID":"6e38970c-7160-4875-86b9-90b48d1a0c9e","Type":"ContainerDied","Data":"e16891a42789c6c7bfd33015ef4e413816cd7382f02d0f97007a988a54c89f4f"} Nov 03 09:08:47.377953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.377765 5015 scope.go:122] "RemoveContainer" containerID="b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de" Nov 03 09:08:47.381439 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.381373 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" event={"ID":"c297a297-b068-459e-a32c-c56bdb9b317f","Type":"ContainerStarted","Data":"dcb637c427813ffd57b3e72d2cdc6eefd6ed34dd046f539a12bec9076139d5bd"} Nov 03 09:08:47.391819 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.391793 5015 scope.go:122] "RemoveContainer" containerID="b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de" Nov 03 09:08:47.392139 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:08:47.392119 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de\": container with ID starting with b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de not found: ID does not exist" containerID="b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de" Nov 03 09:08:47.392211 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.392149 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de"} err="failed to get container status \"b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de\": rpc error: code = NotFound desc = could not find container \"b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de\": container with ID starting with b6760647615a10293dac6df17c48051d0070072fe5bf2f3b8b7ed1f077b186de not found: ID does not exist" Nov 03 09:08:47.422772 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.422705 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e38970c-7160-4875-86b9-90b48d1a0c9e-kube-api-access-g68tt\" (UniqueName: \"kubernetes.io/projected/6e38970c-7160-4875-86b9-90b48d1a0c9e-kube-api-access-g68tt\") pod \"6e38970c-7160-4875-86b9-90b48d1a0c9e\" (UID: \"6e38970c-7160-4875-86b9-90b48d1a0c9e\") " Nov 03 09:08:47.422915 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.422853 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e38970c-7160-4875-86b9-90b48d1a0c9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e38970c-7160-4875-86b9-90b48d1a0c9e-plugins-config-volume\") pod \"6e38970c-7160-4875-86b9-90b48d1a0c9e\" (UID: \"6e38970c-7160-4875-86b9-90b48d1a0c9e\") " Nov 03 09:08:47.423313 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.423286 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e38970c-7160-4875-86b9-90b48d1a0c9e-plugins-config-volume" pod "6e38970c-7160-4875-86b9-90b48d1a0c9e" (UID: "6e38970c-7160-4875-86b9-90b48d1a0c9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:47.425005 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.424986 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e38970c-7160-4875-86b9-90b48d1a0c9e-kube-api-access-g68tt" pod "6e38970c-7160-4875-86b9-90b48d1a0c9e" (UID: "6e38970c-7160-4875-86b9-90b48d1a0c9e"). InnerVolumeSpecName "kube-api-access-g68tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:47.432801 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.432766 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8"] Nov 03 09:08:47.432897 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:08:47.432808 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a24a04f_8d4f_4cbe_b131_78aa27d593c1.slice/crio-be7cee2a0d075bf9d0844fdcbebf15e84c95cdb5269bbf69f0420030376a4a89 WatchSource:0}: Error finding container be7cee2a0d075bf9d0844fdcbebf15e84c95cdb5269bbf69f0420030376a4a89: Status 404 returned error can't find the container with id be7cee2a0d075bf9d0844fdcbebf15e84c95cdb5269bbf69f0420030376a4a89 Nov 03 09:08:47.525122 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.525088 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g68tt\" (UniqueName: \"kubernetes.io/projected/6e38970c-7160-4875-86b9-90b48d1a0c9e-kube-api-access-g68tt\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:47.525256 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.525124 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e38970c-7160-4875-86b9-90b48d1a0c9e-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:47.802444 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.802405 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8"] Nov 03 09:08:47.807883 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:47.807854 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-59ff59596c-6n7k8"] Nov 03 09:08:48.392551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:48.392514 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" event={"ID":"9a24a04f-8d4f-4cbe-b131-78aa27d593c1","Type":"ContainerStarted","Data":"a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028"} Nov 03 09:08:48.393033 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:48.392601 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" event={"ID":"9a24a04f-8d4f-4cbe-b131-78aa27d593c1","Type":"ContainerStarted","Data":"be7cee2a0d075bf9d0844fdcbebf15e84c95cdb5269bbf69f0420030376a4a89"} Nov 03 09:08:48.393033 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:48.392754 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:08:48.395147 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:48.395119 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" event={"ID":"c297a297-b068-459e-a32c-c56bdb9b317f","Type":"ContainerStarted","Data":"1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be"} Nov 03 09:08:48.395277 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:48.395251 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:08:48.511715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:48.511660 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" podStartSLOduration=2.511646071 podStartE2EDuration="2.511646071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:08:48.509156621 +0000 UTC m=+1761.434098857" watchObservedRunningTime="2026-11-03 09:08:48.511646071 +0000 UTC m=+1761.436588288" Nov 03 09:08:48.626577 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:48.626517 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" podStartSLOduration=2.626461568 podStartE2EDuration="2.626461568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:08:48.625550269 +0000 UTC m=+1761.550492504" watchObservedRunningTime="2026-11-03 09:08:48.626461568 +0000 UTC m=+1761.551403786" Nov 03 09:08:49.205774 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:49.205743 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e38970c-7160-4875-86b9-90b48d1a0c9e" path="/var/lib/kubelet/pods/6e38970c-7160-4875-86b9-90b48d1a0c9e/volumes" Nov 03 09:08:49.216762 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:49.216738 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:08:49.216904 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:49.216799 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:08:51.500543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.498286 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l"] Nov 03 09:08:51.500543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.499525 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6e38970c-7160-4875-86b9-90b48d1a0c9e" containerName="payload-processing" Nov 03 09:08:51.500543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.499549 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e38970c-7160-4875-86b9-90b48d1a0c9e" containerName="payload-processing" Nov 03 09:08:51.500543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.499772 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e38970c-7160-4875-86b9-90b48d1a0c9e" containerName="payload-processing" Nov 03 09:08:51.505230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.505205 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:08:51.556290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.556259 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l"] Nov 03 09:08:51.598778 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.598731 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l\" (UID: \"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:08:51.598991 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.598863 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwlgx\" (UniqueName: \"kubernetes.io/projected/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-kube-api-access-vwlgx\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l\" (UID: \"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:08:51.618507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.615779 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c32d41\"" Nov 03 09:08:51.618507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.615948 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c32d41-dockercfg-mw6d9\"" Nov 03 09:08:51.672187 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.672154 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj"] Nov 03 09:08:51.678827 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.678804 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:08:51.700085 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.700057 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj"] Nov 03 09:08:51.700765 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.700737 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l\" (UID: \"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:08:51.701001 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.700974 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwlgx\" (UniqueName: \"kubernetes.io/projected/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-kube-api-access-vwlgx\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l\" (UID: \"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:08:51.707985 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.707957 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l\" (UID: \"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:08:51.785563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.773757 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwlgx\" (UniqueName: \"kubernetes.io/projected/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-kube-api-access-vwlgx\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l\" (UID: \"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:08:51.803050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.802990 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj\" (UID: \"ea42e6fc-bfdc-4c19-8f4b-202be0354aab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:08:51.803722 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.803655 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lphkt\" (UniqueName: \"kubernetes.io/projected/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-kube-api-access-lphkt\") pod \"payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj\" (UID: \"ea42e6fc-bfdc-4c19-8f4b-202be0354aab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:08:51.821309 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.821280 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:08:51.911337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.911305 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj\" (UID: \"ea42e6fc-bfdc-4c19-8f4b-202be0354aab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:08:51.911693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.911671 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lphkt\" (UniqueName: \"kubernetes.io/projected/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-kube-api-access-lphkt\") pod \"payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj\" (UID: \"ea42e6fc-bfdc-4c19-8f4b-202be0354aab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:08:51.912818 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.912800 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj\" (UID: \"ea42e6fc-bfdc-4c19-8f4b-202be0354aab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:08:51.973409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:51.973371 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lphkt\" (UniqueName: \"kubernetes.io/projected/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-kube-api-access-lphkt\") pod \"payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj\" (UID: \"ea42e6fc-bfdc-4c19-8f4b-202be0354aab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:08:52.032683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:52.032653 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:08:52.220954 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:52.220925 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l"] Nov 03 09:08:52.221224 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:08:52.221197 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33886406_dd6c_4ce2_8e2c_e0ec3b3b8e0c.slice/crio-25819095c873cfa622193802b528ab16c8298f171773957696bfb2b8d54fd90b WatchSource:0}: Error finding container 25819095c873cfa622193802b528ab16c8298f171773957696bfb2b8d54fd90b: Status 404 returned error can't find the container with id 25819095c873cfa622193802b528ab16c8298f171773957696bfb2b8d54fd90b Nov 03 09:08:52.422766 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:08:52.422732 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea42e6fc_bfdc_4c19_8f4b_202be0354aab.slice/crio-0dae2894e8b1f1afc9621c5d1748bc9fe46a84f6205b21436684c3c20726d9a5 WatchSource:0}: Error finding container 0dae2894e8b1f1afc9621c5d1748bc9fe46a84f6205b21436684c3c20726d9a5: Status 404 returned error can't find the container with id 0dae2894e8b1f1afc9621c5d1748bc9fe46a84f6205b21436684c3c20726d9a5 Nov 03 09:08:52.422891 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:52.422842 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj"] Nov 03 09:08:52.475660 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:52.475626 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" event={"ID":"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c","Type":"ContainerStarted","Data":"23ac607b9e524be60d9a9a0e93856357e79d31aa8c89b93d0b1b5a50f44a3445"} Nov 03 09:08:52.475790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:52.475684 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" event={"ID":"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c","Type":"ContainerStarted","Data":"25819095c873cfa622193802b528ab16c8298f171773957696bfb2b8d54fd90b"} Nov 03 09:08:52.476140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:52.476114 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:08:52.477191 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:52.477164 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" event={"ID":"ea42e6fc-bfdc-4c19-8f4b-202be0354aab","Type":"ContainerStarted","Data":"0dae2894e8b1f1afc9621c5d1748bc9fe46a84f6205b21436684c3c20726d9a5"} Nov 03 09:08:52.598360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:52.598235 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" podStartSLOduration=1.598218363 podStartE2EDuration="1.598218363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:08:52.591665201 +0000 UTC m=+1765.516607418" watchObservedRunningTime="2026-11-03 09:08:52.598218363 +0000 UTC m=+1765.523160582" Nov 03 09:08:53.488681 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:53.488646 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" event={"ID":"ea42e6fc-bfdc-4c19-8f4b-202be0354aab","Type":"ContainerStarted","Data":"106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70"} Nov 03 09:08:53.630510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:53.626968 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" podStartSLOduration=2.626947182 podStartE2EDuration="2.626947182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:08:53.614921311 +0000 UTC m=+1766.539863528" watchObservedRunningTime="2026-11-03 09:08:53.626947182 +0000 UTC m=+1766.551889399" Nov 03 09:08:54.497217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:54.497181 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:08:59.407630 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:59.407596 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:08:59.408131 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:08:59.407676 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:09:00.883365 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:00.883329 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8"] Nov 03 09:09:01.000067 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:01.000017 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf"] Nov 03 09:09:03.492204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:03.492172 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:09:05.507674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:05.507641 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:09:05.884040 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:05.883948 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" podUID="06fca9f0-42e9-4164-9b25-8226f5644f12" containerName="payload-processing" containerID="cri-o://334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48" gracePeriod=25 Nov 03 09:09:06.000881 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.000832 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" podUID="ba4cfbfa-394c-4ce2-bf0e-327e195c0637" containerName="payload-pre-processing" containerID="cri-o://05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7" gracePeriod=25 Nov 03 09:09:06.005634 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:09:06.005206 5015 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fca9f0_42e9_4164_9b25_8226f5644f12.slice/crio-conmon-334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48.scope\": RecentStats: unable to find data in memory cache]" Nov 03 09:09:06.276540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.263146 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:09:06.334669 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.334636 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06fca9f0-42e9-4164-9b25-8226f5644f12-kube-api-access-scjct\" (UniqueName: \"kubernetes.io/projected/06fca9f0-42e9-4164-9b25-8226f5644f12-kube-api-access-scjct\") pod \"06fca9f0-42e9-4164-9b25-8226f5644f12\" (UID: \"06fca9f0-42e9-4164-9b25-8226f5644f12\") " Nov 03 09:09:06.334868 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.334848 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06fca9f0-42e9-4164-9b25-8226f5644f12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fca9f0-42e9-4164-9b25-8226f5644f12-plugins-config-volume\") pod \"06fca9f0-42e9-4164-9b25-8226f5644f12\" (UID: \"06fca9f0-42e9-4164-9b25-8226f5644f12\") " Nov 03 09:09:06.335245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.335218 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06fca9f0-42e9-4164-9b25-8226f5644f12-plugins-config-volume" pod "06fca9f0-42e9-4164-9b25-8226f5644f12" (UID: "06fca9f0-42e9-4164-9b25-8226f5644f12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:09:06.335797 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.335641 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fca9f0-42e9-4164-9b25-8226f5644f12-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:06.337136 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.337095 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fca9f0-42e9-4164-9b25-8226f5644f12-kube-api-access-scjct" pod "06fca9f0-42e9-4164-9b25-8226f5644f12" (UID: "06fca9f0-42e9-4164-9b25-8226f5644f12"). InnerVolumeSpecName "kube-api-access-scjct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:09:06.349234 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.349207 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:09:06.436709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.436681 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-kube-api-access-nlz98\" (UniqueName: \"kubernetes.io/projected/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-kube-api-access-nlz98\") pod \"ba4cfbfa-394c-4ce2-bf0e-327e195c0637\" (UID: \"ba4cfbfa-394c-4ce2-bf0e-327e195c0637\") " Nov 03 09:09:06.436909 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.436873 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-plugins-config-volume\") pod \"ba4cfbfa-394c-4ce2-bf0e-327e195c0637\" (UID: \"ba4cfbfa-394c-4ce2-bf0e-327e195c0637\") " Nov 03 09:09:06.437330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.437301 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-plugins-config-volume" pod "ba4cfbfa-394c-4ce2-bf0e-327e195c0637" (UID: "ba4cfbfa-394c-4ce2-bf0e-327e195c0637"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:09:06.437596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.437574 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scjct\" (UniqueName: \"kubernetes.io/projected/06fca9f0-42e9-4164-9b25-8226f5644f12-kube-api-access-scjct\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:06.437687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.437602 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:06.439591 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.439567 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-kube-api-access-nlz98" pod "ba4cfbfa-394c-4ce2-bf0e-327e195c0637" (UID: "ba4cfbfa-394c-4ce2-bf0e-327e195c0637"). InnerVolumeSpecName "kube-api-access-nlz98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:09:06.539356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.539314 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlz98\" (UniqueName: \"kubernetes.io/projected/ba4cfbfa-394c-4ce2-bf0e-327e195c0637-kube-api-access-nlz98\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:06.593095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.593059 5015 generic.go:356] "Generic (PLEG): container finished" podID="06fca9f0-42e9-4164-9b25-8226f5644f12" containerID="334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48" exitCode=0 Nov 03 09:09:06.593282 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.593110 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" Nov 03 09:09:06.593282 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.593141 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" event={"ID":"06fca9f0-42e9-4164-9b25-8226f5644f12","Type":"ContainerDied","Data":"334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48"} Nov 03 09:09:06.593282 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.593196 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8" event={"ID":"06fca9f0-42e9-4164-9b25-8226f5644f12","Type":"ContainerDied","Data":"a99797349e8c25cd59deb3783ef5ba53ad2083056138781b624b6c2f67dad40f"} Nov 03 09:09:06.593282 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.593222 5015 scope.go:122] "RemoveContainer" containerID="334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48" Nov 03 09:09:06.594845 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.594819 5015 generic.go:356] "Generic (PLEG): container finished" podID="ba4cfbfa-394c-4ce2-bf0e-327e195c0637" containerID="05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7" exitCode=0 Nov 03 09:09:06.595155 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.594858 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" Nov 03 09:09:06.595155 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.594959 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" event={"ID":"ba4cfbfa-394c-4ce2-bf0e-327e195c0637","Type":"ContainerDied","Data":"05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7"} Nov 03 09:09:06.595155 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.594993 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf" event={"ID":"ba4cfbfa-394c-4ce2-bf0e-327e195c0637","Type":"ContainerDied","Data":"61ae2111be273a47bbd41b026370c605fa7ca3181c97a3a0a05be72469d0a345"} Nov 03 09:09:06.611115 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.611093 5015 scope.go:122] "RemoveContainer" containerID="334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48" Nov 03 09:09:06.611433 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:09:06.611413 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48\": container with ID starting with 334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48 not found: ID does not exist" containerID="334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48" Nov 03 09:09:06.611518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.611442 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48"} err="failed to get container status \"334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48\": rpc error: code = NotFound desc = could not find container \"334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48\": container with ID starting with 334f310a70b71475c5e91f1ef28eb141fba9bfe773f2534fb26eb6dea34b8d48 not found: ID does not exist" Nov 03 09:09:06.611518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.611462 5015 scope.go:122] "RemoveContainer" containerID="05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7" Nov 03 09:09:06.621889 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.621871 5015 scope.go:122] "RemoveContainer" containerID="05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7" Nov 03 09:09:06.622168 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:09:06.622150 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7\": container with ID starting with 05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7 not found: ID does not exist" containerID="05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7" Nov 03 09:09:06.622236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.622184 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7"} err="failed to get container status \"05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7\": rpc error: code = NotFound desc = could not find container \"05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7\": container with ID starting with 05c923eb9c84bb644397a98a65c723c4263736e2b29c3c9ceb14410a940fd3e7 not found: ID does not exist" Nov 03 09:09:06.764681 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.764642 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8"] Nov 03 09:09:06.769100 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.769071 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eb9cd2-67664fbdf5-qsnd8"] Nov 03 09:09:06.881290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.881207 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf"] Nov 03 09:09:06.886338 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:06.886309 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eb9cd2-6cc64bb967-t5qcf"] Nov 03 09:09:07.206667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:07.206570 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06fca9f0-42e9-4164-9b25-8226f5644f12" path="/var/lib/kubelet/pods/06fca9f0-42e9-4164-9b25-8226f5644f12/volumes" Nov 03 09:09:07.207048 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:07.207034 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4cfbfa-394c-4ce2-bf0e-327e195c0637" path="/var/lib/kubelet/pods/ba4cfbfa-394c-4ce2-bf0e-327e195c0637/volumes" Nov 03 09:09:16.918898 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:09:16.918811 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:09:20.777395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.777361 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4"] Nov 03 09:09:20.778062 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.778048 5015 cpu_manager.go:414] "containerMap: removing container" podUID="06fca9f0-42e9-4164-9b25-8226f5644f12" containerName="payload-processing" Nov 03 09:09:20.778112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.778064 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06fca9f0-42e9-4164-9b25-8226f5644f12" containerName="payload-processing" Nov 03 09:09:20.778112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.778085 5015 cpu_manager.go:414] "containerMap: removing container" podUID="ba4cfbfa-394c-4ce2-bf0e-327e195c0637" containerName="payload-pre-processing" Nov 03 09:09:20.778112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.778090 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4cfbfa-394c-4ce2-bf0e-327e195c0637" containerName="payload-pre-processing" Nov 03 09:09:20.778206 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.778196 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="06fca9f0-42e9-4164-9b25-8226f5644f12" containerName="payload-processing" Nov 03 09:09:20.778206 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.778205 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4cfbfa-394c-4ce2-bf0e-327e195c0637" containerName="payload-pre-processing" Nov 03 09:09:20.781920 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.781905 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:20.849503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.844701 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:20.849503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.844798 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hsp\" (UniqueName: \"kubernetes.io/projected/8d46a006-7770-41da-9644-5c70213f8dd7-kube-api-access-b7hsp\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:20.849503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.844844 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-metrics-certs\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:20.849503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.844904 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d46a006-7770-41da-9644-5c70213f8dd7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:20.857599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.857565 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4"] Nov 03 09:09:20.884561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.884527 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4ae3db-serving-cert\"" Nov 03 09:09:20.946177 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.946143 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d46a006-7770-41da-9644-5c70213f8dd7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:20.946350 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.946301 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:20.946458 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.946350 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hsp\" (UniqueName: \"kubernetes.io/projected/8d46a006-7770-41da-9644-5c70213f8dd7-kube-api-access-b7hsp\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:20.946458 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.946385 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-metrics-certs\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:20.947217 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:09:20.946493 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-serving-cert: secret "maas-api-e2e-ipp-a-4ae3db-serving-cert" not found Nov 03 09:09:20.947217 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:09:20.946584 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-maas-api-tls podName:8d46a006-7770-41da-9644-5c70213f8dd7 nodeName:}" failed. No retries permitted until 2026-11-03 09:09:21.446561428 +0000 UTC m=+1794.371503638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-maas-api-tls") pod "maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" (UID: "8d46a006-7770-41da-9644-5c70213f8dd7") : secret "maas-api-e2e-ipp-a-4ae3db-serving-cert" not found Nov 03 09:09:20.947217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.946979 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d46a006-7770-41da-9644-5c70213f8dd7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:20.949939 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:20.949911 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-metrics-certs\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:21.007073 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:21.007046 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hsp\" (UniqueName: \"kubernetes.io/projected/8d46a006-7770-41da-9644-5c70213f8dd7-kube-api-access-b7hsp\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:21.454392 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:21.454353 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:21.457211 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:21.457185 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:21.699651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:21.699619 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:22.087571 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:22.087540 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4"] Nov 03 09:09:22.088815 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:09:22.088787 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d46a006_7770_41da_9644_5c70213f8dd7.slice/crio-816f97611834177d73b5a904c06389ff02a22cd4212024034e89b4d00a5ae4d5 WatchSource:0}: Error finding container 816f97611834177d73b5a904c06389ff02a22cd4212024034e89b4d00a5ae4d5: Status 404 returned error can't find the container with id 816f97611834177d73b5a904c06389ff02a22cd4212024034e89b4d00a5ae4d5 Nov 03 09:09:22.090284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:22.090266 5015 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:09:22.732863 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:22.732824 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" event={"ID":"8d46a006-7770-41da-9644-5c70213f8dd7","Type":"ContainerStarted","Data":"816f97611834177d73b5a904c06389ff02a22cd4212024034e89b4d00a5ae4d5"} Nov 03 09:09:23.743250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:23.743216 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" event={"ID":"8d46a006-7770-41da-9644-5c70213f8dd7","Type":"ContainerStarted","Data":"c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f"} Nov 03 09:09:23.743723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:23.743306 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:23.885798 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:23.885744 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" podStartSLOduration=3.0842831569999998 podStartE2EDuration="3.885728808s" totalImagesPullingTime="801.445651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:09:23.881891737 +0000 UTC m=+1796.806833977" watchObservedRunningTime="2026-11-03 09:09:23.885728808 +0000 UTC m=+1796.810671024" Nov 03 09:09:27.525691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:27.525661 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:09:27.532090 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:27.532065 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:09:29.756895 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:29.756858 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:09:35.388907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:35.387563 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s"] Nov 03 09:09:35.388907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:35.387880 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" podUID="27f6d302-4730-41f8-88ca-410efbf69624" containerName="istio-proxy" containerID="cri-o://d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582" gracePeriod=30 Nov 03 09:09:40.732412 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.732383 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:09:40.826236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826201 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-certs\") pod \"27f6d302-4730-41f8-88ca-410efbf69624\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " Nov 03 09:09:40.826411 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826251 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-kube-api-access-cxmsw\" (UniqueName: \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-kube-api-access-cxmsw\") pod \"27f6d302-4730-41f8-88ca-410efbf69624\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " Nov 03 09:09:40.826411 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826371 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27f6d302-4730-41f8-88ca-410efbf69624-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f6d302-4730-41f8-88ca-410efbf69624-istio-podinfo\") pod \"27f6d302-4730-41f8-88ca-410efbf69624\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " Nov 03 09:09:40.826411 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826391 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-envoy\") pod \"27f6d302-4730-41f8-88ca-410efbf69624\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " Nov 03 09:09:40.826611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826422 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-istio-token\" (UniqueName: \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-istio-token\") pod \"27f6d302-4730-41f8-88ca-410efbf69624\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " Nov 03 09:09:40.826611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826444 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-data\") pod \"27f6d302-4730-41f8-88ca-410efbf69624\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " Nov 03 09:09:40.826611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826579 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-certs" pod "27f6d302-4730-41f8-88ca-410efbf69624" (UID: "27f6d302-4730-41f8-88ca-410efbf69624"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:40.826611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826598 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-socket\") pod \"27f6d302-4730-41f8-88ca-410efbf69624\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " Nov 03 09:09:40.826826 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826722 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f6d302-4730-41f8-88ca-410efbf69624-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f6d302-4730-41f8-88ca-410efbf69624-istiod-ca-cert\") pod \"27f6d302-4730-41f8-88ca-410efbf69624\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " Nov 03 09:09:40.826880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826822 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-credential-socket\") pod \"27f6d302-4730-41f8-88ca-410efbf69624\" (UID: \"27f6d302-4730-41f8-88ca-410efbf69624\") " Nov 03 09:09:40.826880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826860 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-socket" pod "27f6d302-4730-41f8-88ca-410efbf69624" (UID: "27f6d302-4730-41f8-88ca-410efbf69624"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:40.826880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.826872 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-data" pod "27f6d302-4730-41f8-88ca-410efbf69624" (UID: "27f6d302-4730-41f8-88ca-410efbf69624"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:40.827164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.827117 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-credential-socket" pod "27f6d302-4730-41f8-88ca-410efbf69624" (UID: "27f6d302-4730-41f8-88ca-410efbf69624"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:40.827414 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.827182 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f6d302-4730-41f8-88ca-410efbf69624-istiod-ca-cert" pod "27f6d302-4730-41f8-88ca-410efbf69624" (UID: "27f6d302-4730-41f8-88ca-410efbf69624"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:09:40.827527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.827509 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:40.827608 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.827528 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:40.827608 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.827538 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f6d302-4730-41f8-88ca-410efbf69624-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:40.827608 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.827547 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:40.827608 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.827555 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:40.828994 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.828968 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-istio-token" pod "27f6d302-4730-41f8-88ca-410efbf69624" (UID: "27f6d302-4730-41f8-88ca-410efbf69624"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:09:40.829102 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.828997 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-kube-api-access-cxmsw" pod "27f6d302-4730-41f8-88ca-410efbf69624" (UID: "27f6d302-4730-41f8-88ca-410efbf69624"). InnerVolumeSpecName "kube-api-access-cxmsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:09:40.829378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.829353 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-envoy" pod "27f6d302-4730-41f8-88ca-410efbf69624" (UID: "27f6d302-4730-41f8-88ca-410efbf69624"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:40.829519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.829371 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27f6d302-4730-41f8-88ca-410efbf69624-istio-podinfo" pod "27f6d302-4730-41f8-88ca-410efbf69624" (UID: "27f6d302-4730-41f8-88ca-410efbf69624"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:09:40.889044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.889011 5015 generic.go:356] "Generic (PLEG): container finished" podID="27f6d302-4730-41f8-88ca-410efbf69624" containerID="d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582" exitCode=0 Nov 03 09:09:40.889202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.889063 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" Nov 03 09:09:40.889202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.889092 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" event={"ID":"27f6d302-4730-41f8-88ca-410efbf69624","Type":"ContainerDied","Data":"d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582"} Nov 03 09:09:40.889202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.889144 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s" event={"ID":"27f6d302-4730-41f8-88ca-410efbf69624","Type":"ContainerDied","Data":"fa6582224c39848a7d857abfee880bd9de120591f3f7cf63a01940cb6801c7d1"} Nov 03 09:09:40.889202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.889165 5015 scope.go:122] "RemoveContainer" containerID="d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582" Nov 03 09:09:40.900710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.900691 5015 scope.go:122] "RemoveContainer" containerID="d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582" Nov 03 09:09:40.901034 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:09:40.901015 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582\": container with ID starting with d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582 not found: ID does not exist" containerID="d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582" Nov 03 09:09:40.901088 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.901042 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582"} err="failed to get container status \"d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582\": rpc error: code = NotFound desc = could not find container \"d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582\": container with ID starting with d08426450fa1cd99a2d87da61e8159b86bfcd74f12f9df0b1e3f4e4d3fd4b582 not found: ID does not exist" Nov 03 09:09:40.929275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.929243 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f6d302-4730-41f8-88ca-410efbf69624-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:40.929275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.929275 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f6d302-4730-41f8-88ca-410efbf69624-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:40.929501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.929290 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:40.929501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:40.929302 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxmsw\" (UniqueName: \"kubernetes.io/projected/27f6d302-4730-41f8-88ca-410efbf69624-kube-api-access-cxmsw\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:41.038575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:41.038526 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s"] Nov 03 09:09:41.044348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:41.044304 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-7gj4s"] Nov 03 09:09:41.206873 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:41.206839 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f6d302-4730-41f8-88ca-410efbf69624" path="/var/lib/kubelet/pods/27f6d302-4730-41f8-88ca-410efbf69624/volumes" Nov 03 09:09:43.108155 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.108126 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm"] Nov 03 09:09:43.108855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.108841 5015 cpu_manager.go:414] "containerMap: removing container" podUID="27f6d302-4730-41f8-88ca-410efbf69624" containerName="istio-proxy" Nov 03 09:09:43.108909 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.108856 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f6d302-4730-41f8-88ca-410efbf69624" containerName="istio-proxy" Nov 03 09:09:43.108993 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.108983 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f6d302-4730-41f8-88ca-410efbf69624" containerName="istio-proxy" Nov 03 09:09:43.114008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.113994 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.168651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.168621 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bcc5ce-serving-cert\"" Nov 03 09:09:43.171667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.171641 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm"] Nov 03 09:09:43.268393 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.268357 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-maas-api-tls\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.268605 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.268407 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-metrics-certs\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.268696 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.268668 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-openshift-service-ca\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.268869 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.268852 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd52r\" (UniqueName: \"kubernetes.io/projected/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-kube-api-access-wd52r\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.374551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.372074 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd52r\" (UniqueName: \"kubernetes.io/projected/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-kube-api-access-wd52r\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.374551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.372301 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-maas-api-tls\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.374551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.372351 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-metrics-certs\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.374551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.372434 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-openshift-service-ca\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.374551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.373630 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-openshift-service-ca\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.374551 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:09:43.374094 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-serving-cert: secret "maas-api-e2e-api-b-bcc5ce-serving-cert" not found Nov 03 09:09:43.374551 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:09:43.374180 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-maas-api-tls podName:cbe60606-a20a-4ce9-bdf9-acbec23c5b3a nodeName:}" failed. No retries permitted until 2026-11-03 09:09:43.874159648 +0000 UTC m=+1816.799101861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-maas-api-tls") pod "maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" (UID: "cbe60606-a20a-4ce9-bdf9-acbec23c5b3a") : secret "maas-api-e2e-api-b-bcc5ce-serving-cert" not found Nov 03 09:09:43.379532 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.379498 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-metrics-certs\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.437220 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.437193 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd52r\" (UniqueName: \"kubernetes.io/projected/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-kube-api-access-wd52r\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.880395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.880351 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-maas-api-tls\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:43.883229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:43.883198 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-maas-api-tls\") pod \"maas-api-e2e-api-b-bcc5ce-667994994-qhjxm\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:44.028579 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:44.028537 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:44.449563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:44.441900 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm"] Nov 03 09:09:44.921604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:44.921558 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" event={"ID":"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a","Type":"ContainerStarted","Data":"77065b7f1fc889b03560362d26862247af4e4f4b36f085917c5cdb1baeb85874"} Nov 03 09:09:45.933002 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:45.932959 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" event={"ID":"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a","Type":"ContainerStarted","Data":"c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41"} Nov 03 09:09:45.933408 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:45.933182 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:46.055254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:46.055203 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" podStartSLOduration=2.256892903 podStartE2EDuration="3.055190573s" totalImagesPullingTime="798.29767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:09:46.053183877 +0000 UTC m=+1818.978126097" watchObservedRunningTime="2026-11-03 09:09:46.055190573 +0000 UTC m=+1818.980132789" Nov 03 09:09:46.902893 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:09:46.902854 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:09:51.975255 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:51.975227 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:09:56.060223 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:56.060182 5015 generic.go:356] "Generic (PLEG): container finished" podID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerID="389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49" exitCode=0 Nov 03 09:09:56.060823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:56.060231 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" event={"ID":"11d1b378-6be0-4b3d-95ef-71b82d571d7f","Type":"ContainerDied","Data":"389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49"} Nov 03 09:09:56.060823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:56.060793 5015 scope.go:122] "RemoveContainer" containerID="389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49" Nov 03 09:09:56.062842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:56.062807 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:09:56.062982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:56.062881 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:09:56.062982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:56.062924 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:09:57.074862 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:57.074815 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" event={"ID":"11d1b378-6be0-4b3d-95ef-71b82d571d7f","Type":"ContainerStarted","Data":"e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d"} Nov 03 09:09:57.984039 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:57.983992 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9"] Nov 03 09:09:58.017001 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.016967 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:09:58.017175 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.017114 5015 patch_prober.go:29] interesting pod/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 09:09:58.017175 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.017159 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 09:09:58.097676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.097642 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" podUID="c29c5af7-5cf9-4865-a962-e7ce5f5a8755" containerName="maas-api" containerID="cri-o://a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c" gracePeriod=30 Nov 03 09:09:58.614790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.614765 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:09:58.694688 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.694585 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-openshift-service-ca\") pod \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " Nov 03 09:09:58.694688 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.694649 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-kube-api-access-vrhpx\" (UniqueName: \"kubernetes.io/projected/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-kube-api-access-vrhpx\") pod \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " Nov 03 09:09:58.694943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.694723 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-metrics-certs\") pod \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " Nov 03 09:09:58.694943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.694775 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-maas-api-tls\") pod \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\" (UID: \"c29c5af7-5cf9-4865-a962-e7ce5f5a8755\") " Nov 03 09:09:58.695067 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.695036 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-openshift-service-ca" pod "c29c5af7-5cf9-4865-a962-e7ce5f5a8755" (UID: "c29c5af7-5cf9-4865-a962-e7ce5f5a8755"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:09:58.695274 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.695256 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:58.697192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.697161 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-metrics-certs" pod "c29c5af7-5cf9-4865-a962-e7ce5f5a8755" (UID: "c29c5af7-5cf9-4865-a962-e7ce5f5a8755"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:09:58.697659 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.697638 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-maas-api-tls" pod "c29c5af7-5cf9-4865-a962-e7ce5f5a8755" (UID: "c29c5af7-5cf9-4865-a962-e7ce5f5a8755"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:09:58.697759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.697707 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-kube-api-access-vrhpx" pod "c29c5af7-5cf9-4865-a962-e7ce5f5a8755" (UID: "c29c5af7-5cf9-4865-a962-e7ce5f5a8755"). InnerVolumeSpecName "kube-api-access-vrhpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:09:58.796150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.796103 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrhpx\" (UniqueName: \"kubernetes.io/projected/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-kube-api-access-vrhpx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:58.796150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.796148 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:58.796150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:58.796159 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29c5af7-5cf9-4865-a962-e7ce5f5a8755-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:59.023497 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:59.022887 5015 patch_prober.go:29] interesting pod/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 09:09:59.023497 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:59.022951 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 09:09:59.127728 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:59.127670 5015 generic.go:356] "Generic (PLEG): container finished" podID="c29c5af7-5cf9-4865-a962-e7ce5f5a8755" containerID="a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c" exitCode=0 Nov 03 09:09:59.128313 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:59.127730 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" Nov 03 09:09:59.128393 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:59.127757 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" event={"ID":"c29c5af7-5cf9-4865-a962-e7ce5f5a8755","Type":"ContainerDied","Data":"a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c"} Nov 03 09:09:59.128456 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:59.128432 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9" event={"ID":"c29c5af7-5cf9-4865-a962-e7ce5f5a8755","Type":"ContainerDied","Data":"fa066e3df85985736508a07c1a5475def3ba65bf64edc46e0105cdbc9cdd43e7"} Nov 03 09:09:59.128538 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:59.128492 5015 scope.go:122] "RemoveContainer" containerID="a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c" Nov 03 09:09:59.144104 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:59.144078 5015 scope.go:122] "RemoveContainer" containerID="a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c" Nov 03 09:09:59.144650 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:09:59.144625 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c\": container with ID starting with a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c not found: ID does not exist" containerID="a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c" Nov 03 09:09:59.144806 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:59.144780 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c"} err="failed to get container status \"a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c\": rpc error: code = NotFound desc = could not find container \"a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c\": container with ID starting with a17ae8f17356d61ab4deea69cf963e14b9ed0a844ecc084a670da1e72d2b8e4c not found: ID does not exist" Nov 03 09:09:59.301581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:59.301470 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9"] Nov 03 09:09:59.374355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:09:59.374318 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9cd2-5b54d46cbd-lhkh9"] Nov 03 09:10:00.017755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:00.017704 5015 patch_prober.go:29] interesting pod/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 09:10:00.017978 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:00.017766 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 09:10:01.017344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:01.017314 5015 patch_prober.go:29] interesting pod/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 09:10:01.017816 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:01.017364 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 09:10:01.205537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:01.205503 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29c5af7-5cf9-4865-a962-e7ce5f5a8755" path="/var/lib/kubelet/pods/c29c5af7-5cf9-4865-a962-e7ce5f5a8755/volumes" Nov 03 09:10:02.017295 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:02.017257 5015 patch_prober.go:29] interesting pod/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 09:10:02.017509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:02.017311 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 09:10:03.025819 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:03.025786 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:10:03.027890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:03.027040 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:10:03.028134 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:03.028114 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:10:13.110039 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.109964 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt"] Nov 03 09:10:13.110683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.110666 5015 cpu_manager.go:414] "containerMap: removing container" podUID="c29c5af7-5cf9-4865-a962-e7ce5f5a8755" containerName="maas-api" Nov 03 09:10:13.110736 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.110685 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29c5af7-5cf9-4865-a962-e7ce5f5a8755" containerName="maas-api" Nov 03 09:10:13.110838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.110828 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29c5af7-5cf9-4865-a962-e7ce5f5a8755" containerName="maas-api" Nov 03 09:10:13.115767 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.115749 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:10:13.171175 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.171146 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt"] Nov 03 09:10:13.188267 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.188232 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswvt\" (UniqueName: \"kubernetes.io/projected/c1f0e49a-ace8-41e6-941d-5486094e76f8-kube-api-access-fswvt\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt\" (UID: \"c1f0e49a-ace8-41e6-941d-5486094e76f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:10:13.188608 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.188407 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f0e49a-ace8-41e6-941d-5486094e76f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt\" (UID: \"c1f0e49a-ace8-41e6-941d-5486094e76f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:10:13.216744 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.216710 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-77da2a-dockercfg-tzk5k\"" Nov 03 09:10:13.217080 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.217055 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-77da2a\"" Nov 03 09:10:13.289781 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.289730 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f0e49a-ace8-41e6-941d-5486094e76f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt\" (UID: \"c1f0e49a-ace8-41e6-941d-5486094e76f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:10:13.290390 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.290364 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswvt\" (UniqueName: \"kubernetes.io/projected/c1f0e49a-ace8-41e6-941d-5486094e76f8-kube-api-access-fswvt\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt\" (UID: \"c1f0e49a-ace8-41e6-941d-5486094e76f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:10:13.292427 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.292398 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f0e49a-ace8-41e6-941d-5486094e76f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt\" (UID: \"c1f0e49a-ace8-41e6-941d-5486094e76f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:10:13.293291 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.293269 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt"] Nov 03 09:10:13.374378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.374305 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt"] Nov 03 09:10:13.374606 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.374436 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:10:13.378416 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.378394 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswvt\" (UniqueName: \"kubernetes.io/projected/c1f0e49a-ace8-41e6-941d-5486094e76f8-kube-api-access-fswvt\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt\" (UID: \"c1f0e49a-ace8-41e6-941d-5486094e76f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:10:13.394595 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.394543 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6833a04-dae7-4186-aa13-79f7db830a2d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt\" (UID: \"d6833a04-dae7-4186-aa13-79f7db830a2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:10:13.394741 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.394683 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8x72\" (UniqueName: \"kubernetes.io/projected/d6833a04-dae7-4186-aa13-79f7db830a2d-kube-api-access-h8x72\") pod \"payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt\" (UID: \"d6833a04-dae7-4186-aa13-79f7db830a2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:10:13.429754 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.429727 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:10:13.495893 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.495864 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6833a04-dae7-4186-aa13-79f7db830a2d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt\" (UID: \"d6833a04-dae7-4186-aa13-79f7db830a2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:10:13.496023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.495982 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8x72\" (UniqueName: \"kubernetes.io/projected/d6833a04-dae7-4186-aa13-79f7db830a2d-kube-api-access-h8x72\") pod \"payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt\" (UID: \"d6833a04-dae7-4186-aa13-79f7db830a2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:10:13.496957 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.496932 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6833a04-dae7-4186-aa13-79f7db830a2d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt\" (UID: \"d6833a04-dae7-4186-aa13-79f7db830a2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:10:13.555270 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.555236 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8x72\" (UniqueName: \"kubernetes.io/projected/d6833a04-dae7-4186-aa13-79f7db830a2d-kube-api-access-h8x72\") pod \"payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt\" (UID: \"d6833a04-dae7-4186-aa13-79f7db830a2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:10:13.716568 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.716489 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:10:13.816878 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:10:13.816848 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f0e49a_ace8_41e6_941d_5486094e76f8.slice/crio-288da991d5d80ef55408e04fbcf3eb99bb1357566455952ae7ad41f4cc288706 WatchSource:0}: Error finding container 288da991d5d80ef55408e04fbcf3eb99bb1357566455952ae7ad41f4cc288706: Status 404 returned error can't find the container with id 288da991d5d80ef55408e04fbcf3eb99bb1357566455952ae7ad41f4cc288706 Nov 03 09:10:13.817264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:13.817242 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt"] Nov 03 09:10:14.099246 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:10:14.099217 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6833a04_dae7_4186_aa13_79f7db830a2d.slice/crio-f191c03879b9bfd3edbc66341b26ae57673509fc259cd4a218e32536e38ab252 WatchSource:0}: Error finding container f191c03879b9bfd3edbc66341b26ae57673509fc259cd4a218e32536e38ab252: Status 404 returned error can't find the container with id f191c03879b9bfd3edbc66341b26ae57673509fc259cd4a218e32536e38ab252 Nov 03 09:10:14.099958 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:14.099936 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt"] Nov 03 09:10:14.327080 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:14.327044 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" event={"ID":"d6833a04-dae7-4186-aa13-79f7db830a2d","Type":"ContainerStarted","Data":"39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97"} Nov 03 09:10:14.327554 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:14.327114 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:10:14.327554 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:14.327140 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" event={"ID":"d6833a04-dae7-4186-aa13-79f7db830a2d","Type":"ContainerStarted","Data":"f191c03879b9bfd3edbc66341b26ae57673509fc259cd4a218e32536e38ab252"} Nov 03 09:10:14.328692 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:14.328666 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" event={"ID":"c1f0e49a-ace8-41e6-941d-5486094e76f8","Type":"ContainerStarted","Data":"b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494"} Nov 03 09:10:14.328786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:14.328719 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" event={"ID":"c1f0e49a-ace8-41e6-941d-5486094e76f8","Type":"ContainerStarted","Data":"288da991d5d80ef55408e04fbcf3eb99bb1357566455952ae7ad41f4cc288706"} Nov 03 09:10:14.328786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:14.328749 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:10:14.444720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:14.444604 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" podStartSLOduration=1.444588949 podStartE2EDuration="1.444588949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:10:14.442215386 +0000 UTC m=+1847.367157606" watchObservedRunningTime="2026-11-03 09:10:14.444588949 +0000 UTC m=+1847.369531165" Nov 03 09:10:14.559167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:14.559106 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" podStartSLOduration=1.55909133 podStartE2EDuration="1.55909133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:10:14.557872526 +0000 UTC m=+1847.482814753" watchObservedRunningTime="2026-11-03 09:10:14.55909133 +0000 UTC m=+1847.484033596" Nov 03 09:10:16.903120 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:10:16.903086 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:10:25.362566 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:25.361927 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:10:25.362566 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:25.362268 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:10:30.303628 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:30.303591 5015 patch_prober.go:29] interesting pod/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:10:30.304017 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:30.303647 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:10:31.477522 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:31.477443 5015 generic.go:356] "Generic (PLEG): container finished" podID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerID="8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3" exitCode=0 Nov 03 09:10:31.477996 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:31.477618 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" event={"ID":"665332b4-9c0b-4a43-ab38-2050abc0521c","Type":"ContainerDied","Data":"8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3"} Nov 03 09:10:31.478332 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:31.478305 5015 scope.go:122] "RemoveContainer" containerID="8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3" Nov 03 09:10:31.480378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:31.480337 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:10:31.480492 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:31.480435 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:10:31.480567 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:31.480504 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:10:32.494025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:32.492677 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" event={"ID":"665332b4-9c0b-4a43-ab38-2050abc0521c","Type":"ContainerStarted","Data":"97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0"} Nov 03 09:10:33.067761 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:33.067709 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:10:33.068284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:33.068256 5015 patch_prober.go:29] interesting pod/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 09:10:33.068642 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:33.068596 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 09:10:34.068503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:34.067842 5015 patch_prober.go:29] interesting pod/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 09:10:34.068503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:34.067906 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 09:10:35.068293 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:35.068255 5015 patch_prober.go:29] interesting pod/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 09:10:35.068640 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:35.068610 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 09:10:36.070972 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:36.070933 5015 patch_prober.go:29] interesting pod/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 09:10:36.071657 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:36.071611 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 09:10:36.493601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:36.493566 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q"] Nov 03 09:10:37.079570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:37.067947 5015 patch_prober.go:29] interesting pod/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 09:10:37.079570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:37.068031 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 09:10:38.098894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:38.098859 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:10:38.100980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:38.100952 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:10:38.103257 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:38.103239 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:10:41.494119 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:41.494060 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" podUID="6d22752b-b9f0-45ba-a8aa-1229378bd3ab" containerName="payload-processing" containerID="cri-o://aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47" gracePeriod=25 Nov 03 09:10:41.774389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:41.774362 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:10:41.872990 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:41.872949 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-kube-api-access-pd5ts\" (UniqueName: \"kubernetes.io/projected/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-kube-api-access-pd5ts\") pod \"6d22752b-b9f0-45ba-a8aa-1229378bd3ab\" (UID: \"6d22752b-b9f0-45ba-a8aa-1229378bd3ab\") " Nov 03 09:10:41.873162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:41.873101 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-plugins-config-volume\") pod \"6d22752b-b9f0-45ba-a8aa-1229378bd3ab\" (UID: \"6d22752b-b9f0-45ba-a8aa-1229378bd3ab\") " Nov 03 09:10:41.873530 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:41.873462 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-plugins-config-volume" pod "6d22752b-b9f0-45ba-a8aa-1229378bd3ab" (UID: "6d22752b-b9f0-45ba-a8aa-1229378bd3ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:10:41.873764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:41.873747 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:41.875466 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:41.875445 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-kube-api-access-pd5ts" pod "6d22752b-b9f0-45ba-a8aa-1229378bd3ab" (UID: "6d22752b-b9f0-45ba-a8aa-1229378bd3ab"). InnerVolumeSpecName "kube-api-access-pd5ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:10:41.974692 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:41.974659 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd5ts\" (UniqueName: \"kubernetes.io/projected/6d22752b-b9f0-45ba-a8aa-1229378bd3ab-kube-api-access-pd5ts\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:42.576590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:42.576550 5015 generic.go:356] "Generic (PLEG): container finished" podID="6d22752b-b9f0-45ba-a8aa-1229378bd3ab" containerID="aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47" exitCode=0 Nov 03 09:10:42.577055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:42.576598 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" event={"ID":"6d22752b-b9f0-45ba-a8aa-1229378bd3ab","Type":"ContainerDied","Data":"aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47"} Nov 03 09:10:42.577055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:42.576616 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" Nov 03 09:10:42.577055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:42.576661 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q" event={"ID":"6d22752b-b9f0-45ba-a8aa-1229378bd3ab","Type":"ContainerDied","Data":"55219bb79dee8903b1385d986dafb37389b01f3a71b31a29fda0a020684d4d36"} Nov 03 09:10:42.577055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:42.576689 5015 scope.go:122] "RemoveContainer" containerID="aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47" Nov 03 09:10:42.591016 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:42.590990 5015 scope.go:122] "RemoveContainer" containerID="aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47" Nov 03 09:10:42.591407 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:10:42.591370 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47\": container with ID starting with aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47 not found: ID does not exist" containerID="aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47" Nov 03 09:10:42.591546 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:42.591407 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47"} err="failed to get container status \"aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47\": rpc error: code = NotFound desc = could not find container \"aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47\": container with ID starting with aeedcd3f8792719196e463c5b0bc4d145276be01ee294b3e159dcb10c265ae47 not found: ID does not exist" Nov 03 09:10:42.753289 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:42.753253 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q"] Nov 03 09:10:42.803842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:42.803811 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-587fcc9cc9-wcz6q"] Nov 03 09:10:43.205982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:43.205953 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d22752b-b9f0-45ba-a8aa-1229378bd3ab" path="/var/lib/kubelet/pods/6d22752b-b9f0-45ba-a8aa-1229378bd3ab/volumes" Nov 03 09:10:45.865553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:45.865517 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt"] Nov 03 09:10:45.866413 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:45.866389 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6d22752b-b9f0-45ba-a8aa-1229378bd3ab" containerName="payload-processing" Nov 03 09:10:45.866413 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:45.866413 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d22752b-b9f0-45ba-a8aa-1229378bd3ab" containerName="payload-processing" Nov 03 09:10:45.866671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:45.866655 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d22752b-b9f0-45ba-a8aa-1229378bd3ab" containerName="payload-processing" Nov 03 09:10:45.872049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:45.872024 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:10:45.928984 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:45.928956 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt"] Nov 03 09:10:45.974373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:45.974335 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c78f2b\"" Nov 03 09:10:46.004112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.004071 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9"] Nov 03 09:10:46.051510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.032560 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs6w5\" (UniqueName: \"kubernetes.io/projected/6ce18997-659f-4612-8c93-d660f3a21673-kube-api-access-bs6w5\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt\" (UID: \"6ce18997-659f-4612-8c93-d660f3a21673\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:10:46.051510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.032705 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce18997-659f-4612-8c93-d660f3a21673-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt\" (UID: \"6ce18997-659f-4612-8c93-d660f3a21673\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:10:46.054649 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.054619 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:10:46.114243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.108094 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9"] Nov 03 09:10:46.120076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.114777 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c78f2b-dockercfg-nx8fs\"" Nov 03 09:10:46.141253 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.139560 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce18997-659f-4612-8c93-d660f3a21673-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt\" (UID: \"6ce18997-659f-4612-8c93-d660f3a21673\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:10:46.141253 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.139754 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx2x7\" (UniqueName: \"kubernetes.io/projected/215363c2-4ede-4a60-b066-d9d500f2c9c1-kube-api-access-jx2x7\") pod \"payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9\" (UID: \"215363c2-4ede-4a60-b066-d9d500f2c9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:10:46.141253 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.139840 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs6w5\" (UniqueName: \"kubernetes.io/projected/6ce18997-659f-4612-8c93-d660f3a21673-kube-api-access-bs6w5\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt\" (UID: \"6ce18997-659f-4612-8c93-d660f3a21673\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:10:46.141253 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.139888 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215363c2-4ede-4a60-b066-d9d500f2c9c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9\" (UID: \"215363c2-4ede-4a60-b066-d9d500f2c9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:10:46.141253 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.140830 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce18997-659f-4612-8c93-d660f3a21673-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt\" (UID: \"6ce18997-659f-4612-8c93-d660f3a21673\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:10:46.200261 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.200228 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs6w5\" (UniqueName: \"kubernetes.io/projected/6ce18997-659f-4612-8c93-d660f3a21673-kube-api-access-bs6w5\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt\" (UID: \"6ce18997-659f-4612-8c93-d660f3a21673\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:10:46.240971 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.240938 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2x7\" (UniqueName: \"kubernetes.io/projected/215363c2-4ede-4a60-b066-d9d500f2c9c1-kube-api-access-jx2x7\") pod \"payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9\" (UID: \"215363c2-4ede-4a60-b066-d9d500f2c9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:10:46.241149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.241014 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215363c2-4ede-4a60-b066-d9d500f2c9c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9\" (UID: \"215363c2-4ede-4a60-b066-d9d500f2c9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:10:46.241819 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.241798 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215363c2-4ede-4a60-b066-d9d500f2c9c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9\" (UID: \"215363c2-4ede-4a60-b066-d9d500f2c9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:10:46.300239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.300207 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2x7\" (UniqueName: \"kubernetes.io/projected/215363c2-4ede-4a60-b066-d9d500f2c9c1-kube-api-access-jx2x7\") pod \"payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9\" (UID: \"215363c2-4ede-4a60-b066-d9d500f2c9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:10:46.442884 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.442806 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:10:46.490529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.490367 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:10:46.836218 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.836164 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9"] Nov 03 09:10:46.907035 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:10:46.907000 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:10:46.934575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:46.934544 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt"] Nov 03 09:10:46.935382 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:10:46.935350 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce18997_659f_4612_8c93_d660f3a21673.slice/crio-676663c4ca949fdab943857f412725ea77490de319f4e4438e16f04db0c3855c WatchSource:0}: Error finding container 676663c4ca949fdab943857f412725ea77490de319f4e4438e16f04db0c3855c: Status 404 returned error can't find the container with id 676663c4ca949fdab943857f412725ea77490de319f4e4438e16f04db0c3855c Nov 03 09:10:47.629745 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:47.629707 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" event={"ID":"215363c2-4ede-4a60-b066-d9d500f2c9c1","Type":"ContainerStarted","Data":"a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114"} Nov 03 09:10:47.630030 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:47.629780 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:10:47.630030 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:47.629803 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" event={"ID":"215363c2-4ede-4a60-b066-d9d500f2c9c1","Type":"ContainerStarted","Data":"7f7e8bca4a0c51004e969d6b496628e5602c31b0cdf79ead367524e23d6a9582"} Nov 03 09:10:47.631540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:47.631494 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" event={"ID":"6ce18997-659f-4612-8c93-d660f3a21673","Type":"ContainerStarted","Data":"e55d1c787693fa4b2e6566ca1835ad832420fb8bfa6ce4ccaeacba8c0a09d29b"} Nov 03 09:10:47.631676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:47.631569 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:10:47.631676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:47.631597 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" event={"ID":"6ce18997-659f-4612-8c93-d660f3a21673","Type":"ContainerStarted","Data":"676663c4ca949fdab943857f412725ea77490de319f4e4438e16f04db0c3855c"} Nov 03 09:10:47.749105 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:47.749046 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" podStartSLOduration=2.749029121 podStartE2EDuration="2.749029121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:10:47.747109411 +0000 UTC m=+1880.672051628" watchObservedRunningTime="2026-11-03 09:10:47.749029121 +0000 UTC m=+1880.673971337" Nov 03 09:10:47.866325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:47.866274 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" podStartSLOduration=2.866258718 podStartE2EDuration="2.866258718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:10:47.861912783 +0000 UTC m=+1880.786855034" watchObservedRunningTime="2026-11-03 09:10:47.866258718 +0000 UTC m=+1880.791200934" Nov 03 09:10:48.617549 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:48.617511 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx"] Nov 03 09:10:48.618007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:48.617742 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" podUID="c5936599-d8f2-4705-9152-81bdabf3319e" containerName="istio-proxy" containerID="cri-o://313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6" gracePeriod=30 Nov 03 09:10:53.889209 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.889140 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:10:53.950911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.950879 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-kube-api-access-phhtt\" (UniqueName: \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-kube-api-access-phhtt\") pod \"c5936599-d8f2-4705-9152-81bdabf3319e\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " Nov 03 09:10:53.951099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.950926 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-istio-token\" (UniqueName: \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-istio-token\") pod \"c5936599-d8f2-4705-9152-81bdabf3319e\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " Nov 03 09:10:53.951099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.951016 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5936599-d8f2-4705-9152-81bdabf3319e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5936599-d8f2-4705-9152-81bdabf3319e-istio-podinfo\") pod \"c5936599-d8f2-4705-9152-81bdabf3319e\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " Nov 03 09:10:53.951099 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.951062 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-envoy\") pod \"c5936599-d8f2-4705-9152-81bdabf3319e\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " Nov 03 09:10:53.951274 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.951159 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-credential-socket\") pod \"c5936599-d8f2-4705-9152-81bdabf3319e\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " Nov 03 09:10:53.951274 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.951185 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5936599-d8f2-4705-9152-81bdabf3319e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5936599-d8f2-4705-9152-81bdabf3319e-istiod-ca-cert\") pod \"c5936599-d8f2-4705-9152-81bdabf3319e\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " Nov 03 09:10:53.951274 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.951233 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-certs\") pod \"c5936599-d8f2-4705-9152-81bdabf3319e\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " Nov 03 09:10:53.951274 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.951255 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-socket\") pod \"c5936599-d8f2-4705-9152-81bdabf3319e\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " Nov 03 09:10:53.951876 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.951330 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-data\") pod \"c5936599-d8f2-4705-9152-81bdabf3319e\" (UID: \"c5936599-d8f2-4705-9152-81bdabf3319e\") " Nov 03 09:10:53.951876 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.951558 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-credential-socket" pod "c5936599-d8f2-4705-9152-81bdabf3319e" (UID: "c5936599-d8f2-4705-9152-81bdabf3319e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:53.951876 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.951763 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-certs" pod "c5936599-d8f2-4705-9152-81bdabf3319e" (UID: "c5936599-d8f2-4705-9152-81bdabf3319e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:53.952661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.952498 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:53.952661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.952525 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:53.952661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.952595 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5936599-d8f2-4705-9152-81bdabf3319e-istiod-ca-cert" pod "c5936599-d8f2-4705-9152-81bdabf3319e" (UID: "c5936599-d8f2-4705-9152-81bdabf3319e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:10:53.952661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.952607 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-data" pod "c5936599-d8f2-4705-9152-81bdabf3319e" (UID: "c5936599-d8f2-4705-9152-81bdabf3319e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:53.953061 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.953034 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-socket" pod "c5936599-d8f2-4705-9152-81bdabf3319e" (UID: "c5936599-d8f2-4705-9152-81bdabf3319e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:53.953937 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.953915 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-kube-api-access-phhtt" pod "c5936599-d8f2-4705-9152-81bdabf3319e" (UID: "c5936599-d8f2-4705-9152-81bdabf3319e"). InnerVolumeSpecName "kube-api-access-phhtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:10:53.954254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.954222 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-istio-token" pod "c5936599-d8f2-4705-9152-81bdabf3319e" (UID: "c5936599-d8f2-4705-9152-81bdabf3319e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:10:53.954358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.954278 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5936599-d8f2-4705-9152-81bdabf3319e-istio-podinfo" pod "c5936599-d8f2-4705-9152-81bdabf3319e" (UID: "c5936599-d8f2-4705-9152-81bdabf3319e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:10:53.955755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:53.955718 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-envoy" pod "c5936599-d8f2-4705-9152-81bdabf3319e" (UID: "c5936599-d8f2-4705-9152-81bdabf3319e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:54.067842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.060049 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5936599-d8f2-4705-9152-81bdabf3319e-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.067842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.060269 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.067842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.060299 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5936599-d8f2-4705-9152-81bdabf3319e-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.067842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.060313 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.067842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.060326 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5936599-d8f2-4705-9152-81bdabf3319e-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.067842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.060339 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phhtt\" (UniqueName: \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-kube-api-access-phhtt\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.067842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.060357 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5936599-d8f2-4705-9152-81bdabf3319e-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.701354 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.701314 5015 generic.go:356] "Generic (PLEG): container finished" podID="c5936599-d8f2-4705-9152-81bdabf3319e" containerID="313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6" exitCode=0 Nov 03 09:10:54.701569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.701376 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" Nov 03 09:10:54.701569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.701399 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" event={"ID":"c5936599-d8f2-4705-9152-81bdabf3319e","Type":"ContainerDied","Data":"313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6"} Nov 03 09:10:54.701569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.701464 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx" event={"ID":"c5936599-d8f2-4705-9152-81bdabf3319e","Type":"ContainerDied","Data":"15a1d1ad541bf89af24c29bc4e7b15fcd95f0ae179d6867cc16465ef0da53272"} Nov 03 09:10:54.701569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.701535 5015 scope.go:122] "RemoveContainer" containerID="313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6" Nov 03 09:10:54.713377 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.713359 5015 scope.go:122] "RemoveContainer" containerID="313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6" Nov 03 09:10:54.713680 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:10:54.713660 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6\": container with ID starting with 313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6 not found: ID does not exist" containerID="313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6" Nov 03 09:10:54.713760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.713688 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6"} err="failed to get container status \"313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6\": rpc error: code = NotFound desc = could not find container \"313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6\": container with ID starting with 313de32fa330740e3c3a3aba324f0fdff08de088ba6ca9896fb20c93a19b4ec6 not found: ID does not exist" Nov 03 09:10:54.923218 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.923184 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx"] Nov 03 09:10:54.975760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:54.975728 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-7c2gx"] Nov 03 09:10:55.242912 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:55.242820 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5936599-d8f2-4705-9152-81bdabf3319e" path="/var/lib/kubelet/pods/c5936599-d8f2-4705-9152-81bdabf3319e/volumes" Nov 03 09:10:58.642356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:58.642322 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:10:58.642795 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:10:58.642403 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:11:08.611163 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:08.611126 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g"] Nov 03 09:11:11.090786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:11.090755 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg"] Nov 03 09:11:11.211315 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:11.211284 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7"] Nov 03 09:11:13.558681 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:13.558643 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj"] Nov 03 09:11:13.612001 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:13.611958 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" podUID="be58420c-fc67-4845-9088-65718c449087" containerName="payload-processing" containerID="cri-o://22aefdaeec7262c3409adb37981cf5774698cbb7a2c090774b31af28c6e6a695" gracePeriod=25 Nov 03 09:11:13.672980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:13.672948 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l"] Nov 03 09:11:13.867557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:13.864274 5015 generic.go:356] "Generic (PLEG): container finished" podID="be58420c-fc67-4845-9088-65718c449087" containerID="22aefdaeec7262c3409adb37981cf5774698cbb7a2c090774b31af28c6e6a695" exitCode=0 Nov 03 09:11:13.867557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:13.864423 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" event={"ID":"be58420c-fc67-4845-9088-65718c449087","Type":"ContainerDied","Data":"22aefdaeec7262c3409adb37981cf5774698cbb7a2c090774b31af28c6e6a695"} Nov 03 09:11:13.924351 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:13.924321 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:11:13.982735 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:13.982700 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be58420c-fc67-4845-9088-65718c449087-kube-api-access-4pkpk\" (UniqueName: \"kubernetes.io/projected/be58420c-fc67-4845-9088-65718c449087-kube-api-access-4pkpk\") pod \"be58420c-fc67-4845-9088-65718c449087\" (UID: \"be58420c-fc67-4845-9088-65718c449087\") " Nov 03 09:11:13.982937 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:13.982831 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be58420c-fc67-4845-9088-65718c449087-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be58420c-fc67-4845-9088-65718c449087-plugins-config-volume\") pod \"be58420c-fc67-4845-9088-65718c449087\" (UID: \"be58420c-fc67-4845-9088-65718c449087\") " Nov 03 09:11:13.996102 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:13.983808 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be58420c-fc67-4845-9088-65718c449087-plugins-config-volume" pod "be58420c-fc67-4845-9088-65718c449087" (UID: "be58420c-fc67-4845-9088-65718c449087"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:11:13.996102 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:13.991374 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be58420c-fc67-4845-9088-65718c449087-kube-api-access-4pkpk" pod "be58420c-fc67-4845-9088-65718c449087" (UID: "be58420c-fc67-4845-9088-65718c449087"). InnerVolumeSpecName "kube-api-access-4pkpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:14.093753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:14.093153 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pkpk\" (UniqueName: \"kubernetes.io/projected/be58420c-fc67-4845-9088-65718c449087-kube-api-access-4pkpk\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:14.093753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:14.093194 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be58420c-fc67-4845-9088-65718c449087-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:14.875162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:14.875132 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" Nov 03 09:11:14.875623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:14.875131 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g" event={"ID":"be58420c-fc67-4845-9088-65718c449087","Type":"ContainerDied","Data":"5e00c91b2fdeb0b48870ea4073eed055024cdeca71f728887a0d920bae550393"} Nov 03 09:11:14.875623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:14.875274 5015 scope.go:122] "RemoveContainer" containerID="22aefdaeec7262c3409adb37981cf5774698cbb7a2c090774b31af28c6e6a695" Nov 03 09:11:15.048926 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:15.048882 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g"] Nov 03 09:11:15.128557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:15.128002 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-840559-65698dd65c-8d26g"] Nov 03 09:11:15.207985 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:15.207958 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be58420c-fc67-4845-9088-65718c449087" path="/var/lib/kubelet/pods/be58420c-fc67-4845-9088-65718c449087/volumes" Nov 03 09:11:16.091090 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.091044 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" podUID="485174e2-1192-4b2f-abaf-f42b74c6a557" containerName="payload-processing" containerID="cri-o://f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706" gracePeriod=25 Nov 03 09:11:16.212199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.212164 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" podUID="49cf2c34-2317-4277-bb48-b76e62e75fe1" containerName="payload-pre-processing" containerID="cri-o://54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341" gracePeriod=25 Nov 03 09:11:16.408604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.408556 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:11:16.423930 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.423902 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/485174e2-1192-4b2f-abaf-f42b74c6a557-kube-api-access-vqx8l\" (UniqueName: \"kubernetes.io/projected/485174e2-1192-4b2f-abaf-f42b74c6a557-kube-api-access-vqx8l\") pod \"485174e2-1192-4b2f-abaf-f42b74c6a557\" (UID: \"485174e2-1192-4b2f-abaf-f42b74c6a557\") " Nov 03 09:11:16.424093 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.423964 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/485174e2-1192-4b2f-abaf-f42b74c6a557-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485174e2-1192-4b2f-abaf-f42b74c6a557-plugins-config-volume\") pod \"485174e2-1192-4b2f-abaf-f42b74c6a557\" (UID: \"485174e2-1192-4b2f-abaf-f42b74c6a557\") " Nov 03 09:11:16.424355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.424331 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/485174e2-1192-4b2f-abaf-f42b74c6a557-plugins-config-volume" pod "485174e2-1192-4b2f-abaf-f42b74c6a557" (UID: "485174e2-1192-4b2f-abaf-f42b74c6a557"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:11:16.424529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.424508 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485174e2-1192-4b2f-abaf-f42b74c6a557-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:16.426710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.426685 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485174e2-1192-4b2f-abaf-f42b74c6a557-kube-api-access-vqx8l" pod "485174e2-1192-4b2f-abaf-f42b74c6a557" (UID: "485174e2-1192-4b2f-abaf-f42b74c6a557"). InnerVolumeSpecName "kube-api-access-vqx8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:16.494457 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.494429 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:11:16.525524 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.525429 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cf2c34-2317-4277-bb48-b76e62e75fe1-kube-api-access-vlrcb\" (UniqueName: \"kubernetes.io/projected/49cf2c34-2317-4277-bb48-b76e62e75fe1-kube-api-access-vlrcb\") pod \"49cf2c34-2317-4277-bb48-b76e62e75fe1\" (UID: \"49cf2c34-2317-4277-bb48-b76e62e75fe1\") " Nov 03 09:11:16.526582 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.526538 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49cf2c34-2317-4277-bb48-b76e62e75fe1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cf2c34-2317-4277-bb48-b76e62e75fe1-plugins-config-volume\") pod \"49cf2c34-2317-4277-bb48-b76e62e75fe1\" (UID: \"49cf2c34-2317-4277-bb48-b76e62e75fe1\") " Nov 03 09:11:16.527393 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.527366 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cf2c34-2317-4277-bb48-b76e62e75fe1-plugins-config-volume" pod "49cf2c34-2317-4277-bb48-b76e62e75fe1" (UID: "49cf2c34-2317-4277-bb48-b76e62e75fe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:11:16.527541 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.527446 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqx8l\" (UniqueName: \"kubernetes.io/projected/485174e2-1192-4b2f-abaf-f42b74c6a557-kube-api-access-vqx8l\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:16.528788 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.528749 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cf2c34-2317-4277-bb48-b76e62e75fe1-kube-api-access-vlrcb" pod "49cf2c34-2317-4277-bb48-b76e62e75fe1" (UID: "49cf2c34-2317-4277-bb48-b76e62e75fe1"). InnerVolumeSpecName "kube-api-access-vlrcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:16.629188 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.629110 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlrcb\" (UniqueName: \"kubernetes.io/projected/49cf2c34-2317-4277-bb48-b76e62e75fe1-kube-api-access-vlrcb\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:16.629188 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.629136 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cf2c34-2317-4277-bb48-b76e62e75fe1-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:16.897760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.897657 5015 generic.go:356] "Generic (PLEG): container finished" podID="49cf2c34-2317-4277-bb48-b76e62e75fe1" containerID="54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341" exitCode=0 Nov 03 09:11:16.897760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.897707 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" Nov 03 09:11:16.897760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.897738 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" event={"ID":"49cf2c34-2317-4277-bb48-b76e62e75fe1","Type":"ContainerDied","Data":"54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341"} Nov 03 09:11:16.898058 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.897805 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7" event={"ID":"49cf2c34-2317-4277-bb48-b76e62e75fe1","Type":"ContainerDied","Data":"8d77716c0bf35b7386d4d8077401209d5920857f69580f295d739daa355bc0e5"} Nov 03 09:11:16.898058 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.897836 5015 scope.go:122] "RemoveContainer" containerID="54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341" Nov 03 09:11:16.899782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.899747 5015 generic.go:356] "Generic (PLEG): container finished" podID="485174e2-1192-4b2f-abaf-f42b74c6a557" containerID="f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706" exitCode=0 Nov 03 09:11:16.899891 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.899797 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" Nov 03 09:11:16.899948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.899892 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" event={"ID":"485174e2-1192-4b2f-abaf-f42b74c6a557","Type":"ContainerDied","Data":"f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706"} Nov 03 09:11:16.899948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.899937 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg" event={"ID":"485174e2-1192-4b2f-abaf-f42b74c6a557","Type":"ContainerDied","Data":"029e1cc9fa3012a97d3e9a8af553687ccad191a0d25330e8a93e243cf5a9d740"} Nov 03 09:11:16.910497 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:11:16.910156 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:11:16.911275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.911255 5015 scope.go:122] "RemoveContainer" containerID="54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341" Nov 03 09:11:16.911901 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:11:16.911615 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341\": container with ID starting with 54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341 not found: ID does not exist" containerID="54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341" Nov 03 09:11:16.911901 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.911657 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341"} err="failed to get container status \"54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341\": rpc error: code = NotFound desc = could not find container \"54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341\": container with ID starting with 54183d8d40bfaf998f9099d38f35796939744f8e962f2d506c89e4c3830ed341 not found: ID does not exist" Nov 03 09:11:16.911901 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.911681 5015 scope.go:122] "RemoveContainer" containerID="f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706" Nov 03 09:11:16.924077 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.924058 5015 scope.go:122] "RemoveContainer" containerID="f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706" Nov 03 09:11:16.924367 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:11:16.924349 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706\": container with ID starting with f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706 not found: ID does not exist" containerID="f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706" Nov 03 09:11:16.924409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:16.924383 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706"} err="failed to get container status \"f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706\": rpc error: code = NotFound desc = could not find container \"f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706\": container with ID starting with f1de19c68acf90eac8a2e1f53f2476137ade0e7c8fece4709bec742cde1f1706 not found: ID does not exist" Nov 03 09:11:17.069390 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:17.069356 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7"] Nov 03 09:11:17.120078 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:17.120049 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b0c653f4-889fc7759-c62s7"] Nov 03 09:11:17.192946 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:17.192861 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg"] Nov 03 09:11:17.196374 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:17.196347 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b0c653f4-6c6bf7d5df-qnxdg"] Nov 03 09:11:17.206558 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:17.206520 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="485174e2-1192-4b2f-abaf-f42b74c6a557" path="/var/lib/kubelet/pods/485174e2-1192-4b2f-abaf-f42b74c6a557/volumes" Nov 03 09:11:17.206982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:17.206967 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49cf2c34-2317-4277-bb48-b76e62e75fe1" path="/var/lib/kubelet/pods/49cf2c34-2317-4277-bb48-b76e62e75fe1/volumes" Nov 03 09:11:18.559086 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.559050 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" podUID="ea42e6fc-bfdc-4c19-8f4b-202be0354aab" containerName="payload-processing" containerID="cri-o://106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70" gracePeriod=25 Nov 03 09:11:18.673929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.673892 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" podUID="33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c" containerName="payload-pre-processing" containerID="cri-o://23ac607b9e524be60d9a9a0e93856357e79d31aa8c89b93d0b1b5a50f44a3445" gracePeriod=25 Nov 03 09:11:18.842304 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.842263 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:11:18.927981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.927947 5015 generic.go:356] "Generic (PLEG): container finished" podID="33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c" containerID="23ac607b9e524be60d9a9a0e93856357e79d31aa8c89b93d0b1b5a50f44a3445" exitCode=0 Nov 03 09:11:18.928192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.928012 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" event={"ID":"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c","Type":"ContainerDied","Data":"23ac607b9e524be60d9a9a0e93856357e79d31aa8c89b93d0b1b5a50f44a3445"} Nov 03 09:11:18.929742 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.929717 5015 generic.go:356] "Generic (PLEG): container finished" podID="ea42e6fc-bfdc-4c19-8f4b-202be0354aab" containerID="106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70" exitCode=0 Nov 03 09:11:18.929843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.929774 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" Nov 03 09:11:18.929924 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.929845 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" event={"ID":"ea42e6fc-bfdc-4c19-8f4b-202be0354aab","Type":"ContainerDied","Data":"106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70"} Nov 03 09:11:18.929924 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.929914 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj" event={"ID":"ea42e6fc-bfdc-4c19-8f4b-202be0354aab","Type":"ContainerDied","Data":"0dae2894e8b1f1afc9621c5d1748bc9fe46a84f6205b21436684c3c20726d9a5"} Nov 03 09:11:18.930038 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.929941 5015 scope.go:122] "RemoveContainer" containerID="106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70" Nov 03 09:11:18.943272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.943255 5015 scope.go:122] "RemoveContainer" containerID="106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70" Nov 03 09:11:18.943575 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:11:18.943553 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70\": container with ID starting with 106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70 not found: ID does not exist" containerID="106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70" Nov 03 09:11:18.943707 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.943586 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70"} err="failed to get container status \"106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70\": rpc error: code = NotFound desc = could not find container \"106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70\": container with ID starting with 106a77e92525edeb1f1769d104b9318602bfffb19f0fdf346f25244306e73d70 not found: ID does not exist" Nov 03 09:11:18.958846 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.958693 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-plugins-config-volume\") pod \"ea42e6fc-bfdc-4c19-8f4b-202be0354aab\" (UID: \"ea42e6fc-bfdc-4c19-8f4b-202be0354aab\") " Nov 03 09:11:18.958846 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.958796 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-kube-api-access-lphkt\" (UniqueName: \"kubernetes.io/projected/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-kube-api-access-lphkt\") pod \"ea42e6fc-bfdc-4c19-8f4b-202be0354aab\" (UID: \"ea42e6fc-bfdc-4c19-8f4b-202be0354aab\") " Nov 03 09:11:18.959615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.959440 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-plugins-config-volume" pod "ea42e6fc-bfdc-4c19-8f4b-202be0354aab" (UID: "ea42e6fc-bfdc-4c19-8f4b-202be0354aab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:11:18.963941 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.963589 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-kube-api-access-lphkt" pod "ea42e6fc-bfdc-4c19-8f4b-202be0354aab" (UID: "ea42e6fc-bfdc-4c19-8f4b-202be0354aab"). InnerVolumeSpecName "kube-api-access-lphkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:18.979760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:18.979734 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:11:19.060578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.060535 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-plugins-config-volume\") pod \"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c\" (UID: \"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c\") " Nov 03 09:11:19.060747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.060594 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-kube-api-access-vwlgx\" (UniqueName: \"kubernetes.io/projected/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-kube-api-access-vwlgx\") pod \"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c\" (UID: \"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c\") " Nov 03 09:11:19.060921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.060896 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-plugins-config-volume" pod "33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c" (UID: "33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:11:19.061169 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.061148 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:19.061269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.061175 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lphkt\" (UniqueName: \"kubernetes.io/projected/ea42e6fc-bfdc-4c19-8f4b-202be0354aab-kube-api-access-lphkt\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:19.061269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.061189 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:19.062982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.062959 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-kube-api-access-vwlgx" pod "33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c" (UID: "33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c"). InnerVolumeSpecName "kube-api-access-vwlgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:19.162652 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.162567 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwlgx\" (UniqueName: \"kubernetes.io/projected/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c-kube-api-access-vwlgx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:19.350148 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.350114 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj"] Nov 03 09:11:19.355538 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.355512 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cdc9cbffc-q8xzj"] Nov 03 09:11:19.941184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.941090 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" event={"ID":"33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c","Type":"ContainerDied","Data":"25819095c873cfa622193802b528ab16c8298f171773957696bfb2b8d54fd90b"} Nov 03 09:11:19.941184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.941118 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l" Nov 03 09:11:19.941184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:19.941167 5015 scope.go:122] "RemoveContainer" containerID="23ac607b9e524be60d9a9a0e93856357e79d31aa8c89b93d0b1b5a50f44a3445" Nov 03 09:11:20.112425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:20.112391 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l"] Nov 03 09:11:20.163329 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:20.163297 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-77558847f9-9vj9l"] Nov 03 09:11:21.204874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:21.204845 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c" path="/var/lib/kubelet/pods/33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c/volumes" Nov 03 09:11:21.205355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:21.205335 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea42e6fc-bfdc-4c19-8f4b-202be0354aab" path="/var/lib/kubelet/pods/ea42e6fc-bfdc-4c19-8f4b-202be0354aab/volumes" Nov 03 09:11:28.443444 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.443409 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv"] Nov 03 09:11:28.443992 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.443673 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" podUID="064c9fc1-8346-4d21-9301-5becefffa0c5" containerName="maas-api" containerID="cri-o://d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a" gracePeriod=30 Nov 03 09:11:28.721094 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.721067 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:11:28.805968 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.805932 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064c9fc1-8346-4d21-9301-5becefffa0c5-kube-api-access-7krzp\" (UniqueName: \"kubernetes.io/projected/064c9fc1-8346-4d21-9301-5becefffa0c5-kube-api-access-7krzp\") pod \"064c9fc1-8346-4d21-9301-5becefffa0c5\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " Nov 03 09:11:28.806168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.806004 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-maas-api-tls\") pod \"064c9fc1-8346-4d21-9301-5becefffa0c5\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " Nov 03 09:11:28.806168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.806164 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/064c9fc1-8346-4d21-9301-5becefffa0c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064c9fc1-8346-4d21-9301-5becefffa0c5-openshift-service-ca\") pod \"064c9fc1-8346-4d21-9301-5becefffa0c5\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " Nov 03 09:11:28.806292 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.806274 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-metrics-certs\") pod \"064c9fc1-8346-4d21-9301-5becefffa0c5\" (UID: \"064c9fc1-8346-4d21-9301-5becefffa0c5\") " Nov 03 09:11:28.806618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.806589 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064c9fc1-8346-4d21-9301-5becefffa0c5-openshift-service-ca" pod "064c9fc1-8346-4d21-9301-5becefffa0c5" (UID: "064c9fc1-8346-4d21-9301-5becefffa0c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:11:28.806932 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.806806 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/064c9fc1-8346-4d21-9301-5becefffa0c5-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:28.808325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.808305 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-maas-api-tls" pod "064c9fc1-8346-4d21-9301-5becefffa0c5" (UID: "064c9fc1-8346-4d21-9301-5becefffa0c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:11:28.808426 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.808367 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064c9fc1-8346-4d21-9301-5becefffa0c5-kube-api-access-7krzp" pod "064c9fc1-8346-4d21-9301-5becefffa0c5" (UID: "064c9fc1-8346-4d21-9301-5becefffa0c5"). InnerVolumeSpecName "kube-api-access-7krzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:28.808602 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.808579 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-metrics-certs" pod "064c9fc1-8346-4d21-9301-5becefffa0c5" (UID: "064c9fc1-8346-4d21-9301-5becefffa0c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:11:28.911948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.911904 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:28.911948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.911947 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7krzp\" (UniqueName: \"kubernetes.io/projected/064c9fc1-8346-4d21-9301-5becefffa0c5-kube-api-access-7krzp\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:28.912231 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:28.911961 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/064c9fc1-8346-4d21-9301-5becefffa0c5-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:29.024931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:29.024898 5015 generic.go:356] "Generic (PLEG): container finished" podID="064c9fc1-8346-4d21-9301-5becefffa0c5" containerID="d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a" exitCode=0 Nov 03 09:11:29.025134 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:29.025001 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" Nov 03 09:11:29.025134 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:29.025006 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" event={"ID":"064c9fc1-8346-4d21-9301-5becefffa0c5","Type":"ContainerDied","Data":"d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a"} Nov 03 09:11:29.025229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:29.025155 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv" event={"ID":"064c9fc1-8346-4d21-9301-5becefffa0c5","Type":"ContainerDied","Data":"b0c9d3e8cf33986d42e4b89af747a5a802bdb9ac347191ac9468b85c8ec25d5c"} Nov 03 09:11:29.025229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:29.025196 5015 scope.go:122] "RemoveContainer" containerID="d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a" Nov 03 09:11:29.044794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:29.044760 5015 scope.go:122] "RemoveContainer" containerID="d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a" Nov 03 09:11:29.045247 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:11:29.045218 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a\": container with ID starting with d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a not found: ID does not exist" containerID="d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a" Nov 03 09:11:29.045360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:29.045265 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a"} err="failed to get container status \"d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a\": rpc error: code = NotFound desc = could not find container \"d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a\": container with ID starting with d7aa3048031f5f7d287dda771ce093515e57d6a10f160a7997a211b57164b52a not found: ID does not exist" Nov 03 09:11:29.196015 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:29.195986 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv"] Nov 03 09:11:29.205993 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:29.205964 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-840559-577b7cd5c9-sg5jv"] Nov 03 09:11:31.205494 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:31.205435 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="064c9fc1-8346-4d21-9301-5becefffa0c5" path="/var/lib/kubelet/pods/064c9fc1-8346-4d21-9301-5becefffa0c5/volumes" Nov 03 09:11:36.390781 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:36.390688 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2"] Nov 03 09:11:37.092686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.092631 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" containerID="cri-o://e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba" gracePeriod=30 Nov 03 09:11:37.095918 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.095892 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs"] Nov 03 09:11:37.096618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096602 5015 cpu_manager.go:414] "containerMap: removing container" podUID="49cf2c34-2317-4277-bb48-b76e62e75fe1" containerName="payload-pre-processing" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096620 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49cf2c34-2317-4277-bb48-b76e62e75fe1" containerName="payload-pre-processing" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096635 5015 cpu_manager.go:414] "containerMap: removing container" podUID="485174e2-1192-4b2f-abaf-f42b74c6a557" containerName="payload-processing" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096642 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="485174e2-1192-4b2f-abaf-f42b74c6a557" containerName="payload-processing" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096663 5015 cpu_manager.go:414] "containerMap: removing container" podUID="ea42e6fc-bfdc-4c19-8f4b-202be0354aab" containerName="payload-processing" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096670 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea42e6fc-bfdc-4c19-8f4b-202be0354aab" containerName="payload-processing" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096681 5015 cpu_manager.go:414] "containerMap: removing container" podUID="33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c" containerName="payload-pre-processing" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096685 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c" containerName="payload-pre-processing" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096699 5015 cpu_manager.go:414] "containerMap: removing container" podUID="064c9fc1-8346-4d21-9301-5becefffa0c5" containerName="maas-api" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096703 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064c9fc1-8346-4d21-9301-5becefffa0c5" containerName="maas-api" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096713 5015 cpu_manager.go:414] "containerMap: removing container" podUID="c5936599-d8f2-4705-9152-81bdabf3319e" containerName="istio-proxy" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096718 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5936599-d8f2-4705-9152-81bdabf3319e" containerName="istio-proxy" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096728 5015 cpu_manager.go:414] "containerMap: removing container" podUID="be58420c-fc67-4845-9088-65718c449087" containerName="payload-processing" Nov 03 09:11:37.096847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096737 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be58420c-fc67-4845-9088-65718c449087" containerName="payload-processing" Nov 03 09:11:37.097590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096863 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="49cf2c34-2317-4277-bb48-b76e62e75fe1" containerName="payload-pre-processing" Nov 03 09:11:37.097590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096874 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="485174e2-1192-4b2f-abaf-f42b74c6a557" containerName="payload-processing" Nov 03 09:11:37.097590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096883 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea42e6fc-bfdc-4c19-8f4b-202be0354aab" containerName="payload-processing" Nov 03 09:11:37.097590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096892 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="33886406-dd6c-4ce2-8e2c-e0ec3b3b8e0c" containerName="payload-pre-processing" Nov 03 09:11:37.097590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096904 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="064c9fc1-8346-4d21-9301-5becefffa0c5" containerName="maas-api" Nov 03 09:11:37.097590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096913 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5936599-d8f2-4705-9152-81bdabf3319e" containerName="istio-proxy" Nov 03 09:11:37.097590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.096919 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="be58420c-fc67-4845-9088-65718c449087" containerName="payload-processing" Nov 03 09:11:37.100708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.100691 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:11:37.185335 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.185297 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjhf\" (UniqueName: \"kubernetes.io/projected/2b3f774c-9781-4bf6-8594-fd0210ad8256-kube-api-access-9bjhf\") pod \"payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs\" (UID: \"2b3f774c-9781-4bf6-8594-fd0210ad8256\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:11:37.185652 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.185626 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3f774c-9781-4bf6-8594-fd0210ad8256-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs\" (UID: \"2b3f774c-9781-4bf6-8594-fd0210ad8256\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:11:37.199956 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.199923 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4ae3db\"" Nov 03 09:11:37.199956 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.199933 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4ae3db-dockercfg-rbvvk\"" Nov 03 09:11:37.205741 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.205714 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs"] Nov 03 09:11:37.286540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.286499 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjhf\" (UniqueName: \"kubernetes.io/projected/2b3f774c-9781-4bf6-8594-fd0210ad8256-kube-api-access-9bjhf\") pod \"payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs\" (UID: \"2b3f774c-9781-4bf6-8594-fd0210ad8256\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:11:37.286726 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.286618 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3f774c-9781-4bf6-8594-fd0210ad8256-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs\" (UID: \"2b3f774c-9781-4bf6-8594-fd0210ad8256\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:11:37.287316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.287296 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3f774c-9781-4bf6-8594-fd0210ad8256-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs\" (UID: \"2b3f774c-9781-4bf6-8594-fd0210ad8256\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:11:37.345806 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.345732 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjhf\" (UniqueName: \"kubernetes.io/projected/2b3f774c-9781-4bf6-8594-fd0210ad8256-kube-api-access-9bjhf\") pod \"payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs\" (UID: \"2b3f774c-9781-4bf6-8594-fd0210ad8256\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:11:37.414189 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.414163 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:11:37.797317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:37.797285 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs"] Nov 03 09:11:37.798064 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:11:37.798038 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3f774c_9781_4bf6_8594_fd0210ad8256.slice/crio-e4a225de316eca7ff75bdb2e0dd5976f279e68a7bce4a2724b89cba153c63896 WatchSource:0}: Error finding container e4a225de316eca7ff75bdb2e0dd5976f279e68a7bce4a2724b89cba153c63896: Status 404 returned error can't find the container with id e4a225de316eca7ff75bdb2e0dd5976f279e68a7bce4a2724b89cba153c63896 Nov 03 09:11:38.101093 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:38.101007 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" event={"ID":"2b3f774c-9781-4bf6-8594-fd0210ad8256","Type":"ContainerStarted","Data":"e4a225de316eca7ff75bdb2e0dd5976f279e68a7bce4a2724b89cba153c63896"} Nov 03 09:11:39.305709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:39.305662 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8"] Nov 03 09:11:39.417389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:39.417334 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6"] Nov 03 09:11:41.125558 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:41.125505 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" event={"ID":"2b3f774c-9781-4bf6-8594-fd0210ad8256","Type":"ContainerStarted","Data":"e68d83ec39ff5850d087287a7046699d85adfb47d13ce4dc799ba4d2042309ac"} Nov 03 09:11:41.126008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:41.125730 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:11:42.394536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.394506 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:11:42.466828 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.466791 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-token\" (UniqueName: \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-token\") pod \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " Nov 03 09:11:42.467014 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.466859 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-certs\") pod \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " Nov 03 09:11:42.467014 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.466961 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istiod-ca-cert\") pod \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " Nov 03 09:11:42.467126 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.467046 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-socket\") pod \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " Nov 03 09:11:42.467182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.467131 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-certs" pod "911aeeac-4f94-4ee1-a6c6-46db527d4f0f" (UID: "911aeeac-4f94-4ee1-a6c6-46db527d4f0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:11:42.467182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.467135 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-data\") pod \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " Nov 03 09:11:42.467290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.467239 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-kube-api-access-nr2bx\" (UniqueName: \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-kube-api-access-nr2bx\") pod \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " Nov 03 09:11:42.467345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.467314 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istiod-ca-cert" pod "911aeeac-4f94-4ee1-a6c6-46db527d4f0f" (UID: "911aeeac-4f94-4ee1-a6c6-46db527d4f0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:11:42.467398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.467349 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-podinfo\") pod \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " Nov 03 09:11:42.467499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.467350 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-socket" pod "911aeeac-4f94-4ee1-a6c6-46db527d4f0f" (UID: "911aeeac-4f94-4ee1-a6c6-46db527d4f0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:11:42.467499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.467375 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-data" pod "911aeeac-4f94-4ee1-a6c6-46db527d4f0f" (UID: "911aeeac-4f94-4ee1-a6c6-46db527d4f0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:11:42.467623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.467451 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-credential-socket\") pod \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " Nov 03 09:11:42.467623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.467568 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-envoy\") pod \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\" (UID: \"911aeeac-4f94-4ee1-a6c6-46db527d4f0f\") " Nov 03 09:11:42.467723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.467676 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-credential-socket" pod "911aeeac-4f94-4ee1-a6c6-46db527d4f0f" (UID: "911aeeac-4f94-4ee1-a6c6-46db527d4f0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:11:42.468158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.468133 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:42.468276 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.468162 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:42.468276 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.468178 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:42.468276 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.468191 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:42.468276 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.468204 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:42.470783 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.470715 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-envoy" pod "911aeeac-4f94-4ee1-a6c6-46db527d4f0f" (UID: "911aeeac-4f94-4ee1-a6c6-46db527d4f0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:11:42.470783 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.470726 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-token" pod "911aeeac-4f94-4ee1-a6c6-46db527d4f0f" (UID: "911aeeac-4f94-4ee1-a6c6-46db527d4f0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:42.470783 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.470730 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-podinfo" pod "911aeeac-4f94-4ee1-a6c6-46db527d4f0f" (UID: "911aeeac-4f94-4ee1-a6c6-46db527d4f0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:11:42.470965 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.470939 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-kube-api-access-nr2bx" pod "911aeeac-4f94-4ee1-a6c6-46db527d4f0f" (UID: "911aeeac-4f94-4ee1-a6c6-46db527d4f0f"). InnerVolumeSpecName "kube-api-access-nr2bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:42.568371 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.568305 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" podStartSLOduration=3.295234135 podStartE2EDuration="5.568286378s" totalImagesPullingTime="2.273052243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:11:41.260095041 +0000 UTC m=+1934.185037265" watchObservedRunningTime="2026-11-03 09:11:42.568286378 +0000 UTC m=+1935.493228596" Nov 03 09:11:42.569803 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.569781 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:42.569803 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.569804 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:42.570000 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.569813 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:42.570000 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:42.569822 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr2bx\" (UniqueName: \"kubernetes.io/projected/911aeeac-4f94-4ee1-a6c6-46db527d4f0f-kube-api-access-nr2bx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:43.146156 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.146110 5015 generic.go:356] "Generic (PLEG): container finished" podID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerID="e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba" exitCode=0 Nov 03 09:11:43.146356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.146195 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" event={"ID":"911aeeac-4f94-4ee1-a6c6-46db527d4f0f","Type":"ContainerDied","Data":"e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba"} Nov 03 09:11:43.146356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.146208 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" Nov 03 09:11:43.146356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.146282 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2" event={"ID":"911aeeac-4f94-4ee1-a6c6-46db527d4f0f","Type":"ContainerDied","Data":"b90fb52997fdde30426578686e29560decef175e07160d043128e48fa73cfe5a"} Nov 03 09:11:43.146356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.146308 5015 scope.go:122] "RemoveContainer" containerID="e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba" Nov 03 09:11:43.158646 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.158618 5015 scope.go:122] "RemoveContainer" containerID="d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339" Nov 03 09:11:43.168682 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.168661 5015 scope.go:122] "RemoveContainer" containerID="e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba" Nov 03 09:11:43.169006 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:11:43.168986 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba\": container with ID starting with e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba not found: ID does not exist" containerID="e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba" Nov 03 09:11:43.169071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.169014 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba"} err="failed to get container status \"e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba\": rpc error: code = NotFound desc = could not find container \"e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba\": container with ID starting with e41d35649c8261ca54f30dc448f4eaa243135b81965e2f0f3367e357b6e4e9ba not found: ID does not exist" Nov 03 09:11:43.169071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.169030 5015 scope.go:122] "RemoveContainer" containerID="d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339" Nov 03 09:11:43.169331 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:11:43.169309 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339\": container with ID starting with d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339 not found: ID does not exist" containerID="d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339" Nov 03 09:11:43.169384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.169339 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339"} err="failed to get container status \"d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339\": rpc error: code = NotFound desc = could not find container \"d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339\": container with ID starting with d8801379fe5540507b7e2b5f5b03c02f6d68fee4bb81ce9ca479ab15151b0339 not found: ID does not exist" Nov 03 09:11:43.368228 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.368192 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2"] Nov 03 09:11:43.371722 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:43.371692 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b0c653f4-openshift-default-5b5557b5bf-8xmb2"] Nov 03 09:11:44.306703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.306659 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" podUID="9a24a04f-8d4f-4cbe-b131-78aa27d593c1" containerName="payload-processing" containerID="cri-o://a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028" gracePeriod=25 Nov 03 09:11:44.418618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.418550 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" podUID="c297a297-b068-459e-a32c-c56bdb9b317f" containerName="payload-pre-processing" containerID="cri-o://1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be" gracePeriod=25 Nov 03 09:11:44.615345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.615297 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:11:44.701659 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.701630 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-kube-api-access-b8x9s\" (UniqueName: \"kubernetes.io/projected/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-kube-api-access-b8x9s\") pod \"9a24a04f-8d4f-4cbe-b131-78aa27d593c1\" (UID: \"9a24a04f-8d4f-4cbe-b131-78aa27d593c1\") " Nov 03 09:11:44.701817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.701797 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-plugins-config-volume\") pod \"9a24a04f-8d4f-4cbe-b131-78aa27d593c1\" (UID: \"9a24a04f-8d4f-4cbe-b131-78aa27d593c1\") " Nov 03 09:11:44.702153 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.702133 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-plugins-config-volume" pod "9a24a04f-8d4f-4cbe-b131-78aa27d593c1" (UID: "9a24a04f-8d4f-4cbe-b131-78aa27d593c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:11:44.702370 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.702344 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:44.704013 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.703989 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-kube-api-access-b8x9s" pod "9a24a04f-8d4f-4cbe-b131-78aa27d593c1" (UID: "9a24a04f-8d4f-4cbe-b131-78aa27d593c1"). InnerVolumeSpecName "kube-api-access-b8x9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:44.707653 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.707635 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:11:44.803841 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.803802 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c297a297-b068-459e-a32c-c56bdb9b317f-kube-api-access-x7rhd\" (UniqueName: \"kubernetes.io/projected/c297a297-b068-459e-a32c-c56bdb9b317f-kube-api-access-x7rhd\") pod \"c297a297-b068-459e-a32c-c56bdb9b317f\" (UID: \"c297a297-b068-459e-a32c-c56bdb9b317f\") " Nov 03 09:11:44.804014 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.803865 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c297a297-b068-459e-a32c-c56bdb9b317f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c297a297-b068-459e-a32c-c56bdb9b317f-plugins-config-volume\") pod \"c297a297-b068-459e-a32c-c56bdb9b317f\" (UID: \"c297a297-b068-459e-a32c-c56bdb9b317f\") " Nov 03 09:11:44.804240 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.804217 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c297a297-b068-459e-a32c-c56bdb9b317f-plugins-config-volume" pod "c297a297-b068-459e-a32c-c56bdb9b317f" (UID: "c297a297-b068-459e-a32c-c56bdb9b317f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:11:44.804384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.804372 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8x9s\" (UniqueName: \"kubernetes.io/projected/9a24a04f-8d4f-4cbe-b131-78aa27d593c1-kube-api-access-b8x9s\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:44.804428 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.804388 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c297a297-b068-459e-a32c-c56bdb9b317f-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:44.806237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.806216 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c297a297-b068-459e-a32c-c56bdb9b317f-kube-api-access-x7rhd" pod "c297a297-b068-459e-a32c-c56bdb9b317f" (UID: "c297a297-b068-459e-a32c-c56bdb9b317f"). InnerVolumeSpecName "kube-api-access-x7rhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:44.905246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:44.905170 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7rhd\" (UniqueName: \"kubernetes.io/projected/c297a297-b068-459e-a32c-c56bdb9b317f-kube-api-access-x7rhd\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:45.170258 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.170151 5015 generic.go:356] "Generic (PLEG): container finished" podID="9a24a04f-8d4f-4cbe-b131-78aa27d593c1" containerID="a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028" exitCode=0 Nov 03 09:11:45.170258 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.170214 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" Nov 03 09:11:45.170512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.170262 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" event={"ID":"9a24a04f-8d4f-4cbe-b131-78aa27d593c1","Type":"ContainerDied","Data":"a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028"} Nov 03 09:11:45.170512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.170341 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8" event={"ID":"9a24a04f-8d4f-4cbe-b131-78aa27d593c1","Type":"ContainerDied","Data":"be7cee2a0d075bf9d0844fdcbebf15e84c95cdb5269bbf69f0420030376a4a89"} Nov 03 09:11:45.170697 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.170373 5015 scope.go:122] "RemoveContainer" containerID="a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028" Nov 03 09:11:45.172111 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.172076 5015 generic.go:356] "Generic (PLEG): container finished" podID="c297a297-b068-459e-a32c-c56bdb9b317f" containerID="1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be" exitCode=0 Nov 03 09:11:45.172246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.172155 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" Nov 03 09:11:45.172246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.172158 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" event={"ID":"c297a297-b068-459e-a32c-c56bdb9b317f","Type":"ContainerDied","Data":"1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be"} Nov 03 09:11:45.172246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.172231 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6" event={"ID":"c297a297-b068-459e-a32c-c56bdb9b317f","Type":"ContainerDied","Data":"dcb637c427813ffd57b3e72d2cdc6eefd6ed34dd046f539a12bec9076139d5bd"} Nov 03 09:11:45.183046 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.183030 5015 scope.go:122] "RemoveContainer" containerID="a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028" Nov 03 09:11:45.183381 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:11:45.183360 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028\": container with ID starting with a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028 not found: ID does not exist" containerID="a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028" Nov 03 09:11:45.183495 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.183387 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028"} err="failed to get container status \"a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028\": rpc error: code = NotFound desc = could not find container \"a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028\": container with ID starting with a141f8ebe8f80124e1c8066879eb81db82a5103de652428aa8d4ac6fd63cd028 not found: ID does not exist" Nov 03 09:11:45.183495 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.183404 5015 scope.go:122] "RemoveContainer" containerID="1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be" Nov 03 09:11:45.194193 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.194168 5015 scope.go:122] "RemoveContainer" containerID="1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be" Nov 03 09:11:45.194448 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:11:45.194429 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be\": container with ID starting with 1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be not found: ID does not exist" containerID="1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be" Nov 03 09:11:45.194525 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.194455 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be"} err="failed to get container status \"1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be\": rpc error: code = NotFound desc = could not find container \"1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be\": container with ID starting with 1e31fbd765f807f4f5dff75ccefab09edb46b92466aca3a2794296890e0cd7be not found: ID does not exist" Nov 03 09:11:45.204513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.204469 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" path="/var/lib/kubelet/pods/911aeeac-4f94-4ee1-a6c6-46db527d4f0f/volumes" Nov 03 09:11:45.341465 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.341433 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6"] Nov 03 09:11:45.391126 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.391088 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcc5ce-6cdc475c9f-wc2v6"] Nov 03 09:11:45.459985 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.459908 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8"] Nov 03 09:11:45.465712 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:45.465683 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcc5ce-858c768c5f-gcqm8"] Nov 03 09:11:46.903017 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:11:46.902981 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:11:47.206641 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:47.206569 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a24a04f-8d4f-4cbe-b131-78aa27d593c1" path="/var/lib/kubelet/pods/9a24a04f-8d4f-4cbe-b131-78aa27d593c1/volumes" Nov 03 09:11:47.206993 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:47.206980 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c297a297-b068-459e-a32c-c56bdb9b317f" path="/var/lib/kubelet/pods/c297a297-b068-459e-a32c-c56bdb9b317f/volumes" Nov 03 09:11:52.136706 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:52.136668 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:11:55.530979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:55.530940 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76"] Nov 03 09:11:55.531397 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:55.531158 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" podUID="8b530646-50c4-4a45-9abe-35b786de0592" containerName="istio-proxy" containerID="cri-o://8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632" gracePeriod=30 Nov 03 09:11:55.583150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:55.583090 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw"] Nov 03 09:11:55.583351 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:55.583319 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" containerID="cri-o://c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46" gracePeriod=30 Nov 03 09:11:55.929915 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:55.929832 5015 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:11:55.929915 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:55.929888 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" podUID="8b530646-50c4-4a45-9abe-35b786de0592" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:11:58.483184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:58.483148 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z85qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:11:58.483667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:11:58.483204 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:12:00.529994 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.529959 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm"] Nov 03 09:12:00.530440 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.530172 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" podUID="cbe60606-a20a-4ce9-bdf9-acbec23c5b3a" containerName="maas-api" containerID="cri-o://c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41" gracePeriod=30 Nov 03 09:12:00.897578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.897544 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 09:12:00.923933 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.923901 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-data\") pod \"8b530646-50c4-4a45-9abe-35b786de0592\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " Nov 03 09:12:00.924128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.923969 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-credential-socket\") pod \"8b530646-50c4-4a45-9abe-35b786de0592\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " Nov 03 09:12:00.924128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924010 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b530646-50c4-4a45-9abe-35b786de0592-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b530646-50c4-4a45-9abe-35b786de0592-istiod-ca-cert\") pod \"8b530646-50c4-4a45-9abe-35b786de0592\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " Nov 03 09:12:00.924128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924046 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-istio-token\" (UniqueName: \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-istio-token\") pod \"8b530646-50c4-4a45-9abe-35b786de0592\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " Nov 03 09:12:00.924128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924099 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-certs\") pod \"8b530646-50c4-4a45-9abe-35b786de0592\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " Nov 03 09:12:00.924343 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924131 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-kube-api-access-nlqck\" (UniqueName: \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-kube-api-access-nlqck\") pod \"8b530646-50c4-4a45-9abe-35b786de0592\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " Nov 03 09:12:00.924343 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924223 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b530646-50c4-4a45-9abe-35b786de0592-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b530646-50c4-4a45-9abe-35b786de0592-istio-podinfo\") pod \"8b530646-50c4-4a45-9abe-35b786de0592\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " Nov 03 09:12:00.924343 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924260 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-data" pod "8b530646-50c4-4a45-9abe-35b786de0592" (UID: "8b530646-50c4-4a45-9abe-35b786de0592"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:00.924343 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924331 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-envoy\") pod \"8b530646-50c4-4a45-9abe-35b786de0592\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " Nov 03 09:12:00.924343 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924324 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-credential-socket" pod "8b530646-50c4-4a45-9abe-35b786de0592" (UID: "8b530646-50c4-4a45-9abe-35b786de0592"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:00.924666 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924362 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-socket\") pod \"8b530646-50c4-4a45-9abe-35b786de0592\" (UID: \"8b530646-50c4-4a45-9abe-35b786de0592\") " Nov 03 09:12:00.924666 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924589 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-certs" pod "8b530646-50c4-4a45-9abe-35b786de0592" (UID: "8b530646-50c4-4a45-9abe-35b786de0592"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:00.924666 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924587 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b530646-50c4-4a45-9abe-35b786de0592-istiod-ca-cert" pod "8b530646-50c4-4a45-9abe-35b786de0592" (UID: "8b530646-50c4-4a45-9abe-35b786de0592"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:12:00.924918 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924894 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-socket" pod "8b530646-50c4-4a45-9abe-35b786de0592" (UID: "8b530646-50c4-4a45-9abe-35b786de0592"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:00.925015 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.924989 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:00.925015 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.925019 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:00.925352 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.925037 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b530646-50c4-4a45-9abe-35b786de0592-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:00.925352 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.925051 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:00.927001 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.926974 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b530646-50c4-4a45-9abe-35b786de0592-istio-podinfo" pod "8b530646-50c4-4a45-9abe-35b786de0592" (UID: "8b530646-50c4-4a45-9abe-35b786de0592"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:12:00.927103 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.927003 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-istio-token" pod "8b530646-50c4-4a45-9abe-35b786de0592" (UID: "8b530646-50c4-4a45-9abe-35b786de0592"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:00.927426 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.927400 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-envoy" pod "8b530646-50c4-4a45-9abe-35b786de0592" (UID: "8b530646-50c4-4a45-9abe-35b786de0592"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:00.927426 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.927417 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-kube-api-access-nlqck" pod "8b530646-50c4-4a45-9abe-35b786de0592" (UID: "8b530646-50c4-4a45-9abe-35b786de0592"). InnerVolumeSpecName "kube-api-access-nlqck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:00.950755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.950732 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 09:12:00.953570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:00.953544 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.029923 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-credential-socket\") pod \"6d138326-bc46-4f60-9606-caa8d164b7cc\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.029993 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-podinfo\") pod \"6d138326-bc46-4f60-9606-caa8d164b7cc\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.030070 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-certs\") pod \"6d138326-bc46-4f60-9606-caa8d164b7cc\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.030149 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-token\" (UniqueName: \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-token\") pod \"6d138326-bc46-4f60-9606-caa8d164b7cc\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.030181 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-maas-api-tls\") pod \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.030273 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-envoy\") pod \"6d138326-bc46-4f60-9606-caa8d164b7cc\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.030300 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-data\") pod \"6d138326-bc46-4f60-9606-caa8d164b7cc\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.030347 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-metrics-certs\") pod \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.030392 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-kube-api-access-7nkhr\" (UniqueName: \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-kube-api-access-7nkhr\") pod \"6d138326-bc46-4f60-9606-caa8d164b7cc\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.030425 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-openshift-service-ca\") pod \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.030463 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d138326-bc46-4f60-9606-caa8d164b7cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d138326-bc46-4f60-9606-caa8d164b7cc-istiod-ca-cert\") pod \"6d138326-bc46-4f60-9606-caa8d164b7cc\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.030520 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-socket\") pod \"6d138326-bc46-4f60-9606-caa8d164b7cc\" (UID: \"6d138326-bc46-4f60-9606-caa8d164b7cc\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.030613 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-kube-api-access-wd52r\" (UniqueName: \"kubernetes.io/projected/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-kube-api-access-wd52r\") pod \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\" (UID: \"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a\") " Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.031219 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlqck\" (UniqueName: \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-kube-api-access-nlqck\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.031241 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b530646-50c4-4a45-9abe-35b786de0592-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.031254 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.031271 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b530646-50c4-4a45-9abe-35b786de0592-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.031285 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b530646-50c4-4a45-9abe-35b786de0592-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.032388 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d138326-bc46-4f60-9606-caa8d164b7cc-istiod-ca-cert" pod "6d138326-bc46-4f60-9606-caa8d164b7cc" (UID: "6d138326-bc46-4f60-9606-caa8d164b7cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.033233 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-openshift-service-ca" pod "cbe60606-a20a-4ce9-bdf9-acbec23c5b3a" (UID: "cbe60606-a20a-4ce9-bdf9-acbec23c5b3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.033567 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-certs" pod "6d138326-bc46-4f60-9606-caa8d164b7cc" (UID: "6d138326-bc46-4f60-9606-caa8d164b7cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.033764 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-credential-socket" pod "6d138326-bc46-4f60-9606-caa8d164b7cc" (UID: "6d138326-bc46-4f60-9606-caa8d164b7cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.035499 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-data" pod "6d138326-bc46-4f60-9606-caa8d164b7cc" (UID: "6d138326-bc46-4f60-9606-caa8d164b7cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:01.057593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.035701 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-socket" pod "6d138326-bc46-4f60-9606-caa8d164b7cc" (UID: "6d138326-bc46-4f60-9606-caa8d164b7cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:01.074406 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.073794 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-token" pod "6d138326-bc46-4f60-9606-caa8d164b7cc" (UID: "6d138326-bc46-4f60-9606-caa8d164b7cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:01.074406 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.074040 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-kube-api-access-wd52r" pod "cbe60606-a20a-4ce9-bdf9-acbec23c5b3a" (UID: "cbe60606-a20a-4ce9-bdf9-acbec23c5b3a"). InnerVolumeSpecName "kube-api-access-wd52r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:01.091834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.091711 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-podinfo" pod "6d138326-bc46-4f60-9606-caa8d164b7cc" (UID: "6d138326-bc46-4f60-9606-caa8d164b7cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:12:01.091834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.091711 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-metrics-certs" pod "cbe60606-a20a-4ce9-bdf9-acbec23c5b3a" (UID: "cbe60606-a20a-4ce9-bdf9-acbec23c5b3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:12:01.091834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.091727 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-maas-api-tls" pod "cbe60606-a20a-4ce9-bdf9-acbec23c5b3a" (UID: "cbe60606-a20a-4ce9-bdf9-acbec23c5b3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:12:01.091834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.091721 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-envoy" pod "6d138326-bc46-4f60-9606-caa8d164b7cc" (UID: "6d138326-bc46-4f60-9606-caa8d164b7cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:01.091834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.091767 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-kube-api-access-7nkhr" pod "6d138326-bc46-4f60-9606-caa8d164b7cc" (UID: "6d138326-bc46-4f60-9606-caa8d164b7cc"). InnerVolumeSpecName "kube-api-access-7nkhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:01.133127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133048 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133079 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133090 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133101 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nkhr\" (UniqueName: \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-kube-api-access-7nkhr\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133113 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133128 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d138326-bc46-4f60-9606-caa8d164b7cc-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133138 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133147 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd52r\" (UniqueName: \"kubernetes.io/projected/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-kube-api-access-wd52r\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133156 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133164 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133172 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d138326-bc46-4f60-9606-caa8d164b7cc-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133181 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d138326-bc46-4f60-9606-caa8d164b7cc-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.133441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.133189 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:01.305762 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.305722 5015 generic.go:356] "Generic (PLEG): container finished" podID="cbe60606-a20a-4ce9-bdf9-acbec23c5b3a" containerID="c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41" exitCode=0 Nov 03 09:12:01.305981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.305784 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" Nov 03 09:12:01.305981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.305809 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" event={"ID":"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a","Type":"ContainerDied","Data":"c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41"} Nov 03 09:12:01.305981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.305892 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm" event={"ID":"cbe60606-a20a-4ce9-bdf9-acbec23c5b3a","Type":"ContainerDied","Data":"77065b7f1fc889b03560362d26862247af4e4f4b36f085917c5cdb1baeb85874"} Nov 03 09:12:01.305981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.305947 5015 scope.go:122] "RemoveContainer" containerID="c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41" Nov 03 09:12:01.307948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.307852 5015 generic.go:356] "Generic (PLEG): container finished" podID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerID="c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46" exitCode=0 Nov 03 09:12:01.307948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.307934 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" event={"ID":"6d138326-bc46-4f60-9606-caa8d164b7cc","Type":"ContainerDied","Data":"c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46"} Nov 03 09:12:01.308091 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.307992 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" event={"ID":"6d138326-bc46-4f60-9606-caa8d164b7cc","Type":"ContainerDied","Data":"c992567ffed35e29dec03cda1daa63d2e4f4b406a8e98ea2bd55e0f76a8e05f4"} Nov 03 09:12:01.308091 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.308066 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw" Nov 03 09:12:01.311957 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.311925 5015 generic.go:356] "Generic (PLEG): container finished" podID="8b530646-50c4-4a45-9abe-35b786de0592" containerID="8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632" exitCode=0 Nov 03 09:12:01.312086 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.311974 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" Nov 03 09:12:01.312086 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.312040 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" event={"ID":"8b530646-50c4-4a45-9abe-35b786de0592","Type":"ContainerDied","Data":"8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632"} Nov 03 09:12:01.312086 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.312082 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76" event={"ID":"8b530646-50c4-4a45-9abe-35b786de0592","Type":"ContainerDied","Data":"09c0ce5ddd15a9554fcb88e9b4e4364b1a60f9c44c7e8d82c331d353348ffeb8"} Nov 03 09:12:01.320292 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.320272 5015 scope.go:122] "RemoveContainer" containerID="c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41" Nov 03 09:12:01.320607 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:12:01.320580 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41\": container with ID starting with c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41 not found: ID does not exist" containerID="c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41" Nov 03 09:12:01.320676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.320618 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41"} err="failed to get container status \"c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41\": rpc error: code = NotFound desc = could not find container \"c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41\": container with ID starting with c086186b7e4ba9216aef3157e9d4acbd12bad66bb0e002e88a36e32e33cd2e41 not found: ID does not exist" Nov 03 09:12:01.320676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.320637 5015 scope.go:122] "RemoveContainer" containerID="c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46" Nov 03 09:12:01.331605 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.331587 5015 scope.go:122] "RemoveContainer" containerID="c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46" Nov 03 09:12:01.331951 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:12:01.331927 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46\": container with ID starting with c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46 not found: ID does not exist" containerID="c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46" Nov 03 09:12:01.332067 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.331956 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46"} err="failed to get container status \"c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46\": rpc error: code = NotFound desc = could not find container \"c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46\": container with ID starting with c707a63292de7ceff367e2379703d50a8638e39f1f3d2c916ecbff45e31dfe46 not found: ID does not exist" Nov 03 09:12:01.332067 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.331978 5015 scope.go:122] "RemoveContainer" containerID="8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632" Nov 03 09:12:01.342894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.342873 5015 scope.go:122] "RemoveContainer" containerID="8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632" Nov 03 09:12:01.343224 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:12:01.343199 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632\": container with ID starting with 8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632 not found: ID does not exist" containerID="8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632" Nov 03 09:12:01.343319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.343232 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632"} err="failed to get container status \"8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632\": rpc error: code = NotFound desc = could not find container \"8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632\": container with ID starting with 8243344a2e0ff40fa6b9ed9d821206b3745f29b0924777aa4b5bcf88b284c632 not found: ID does not exist" Nov 03 09:12:01.478931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.478902 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw"] Nov 03 09:12:01.484256 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.484228 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z85qw"] Nov 03 09:12:01.598080 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.598045 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76"] Nov 03 09:12:01.601692 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.601666 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-6lr76"] Nov 03 09:12:01.712818 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.712782 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm"] Nov 03 09:12:01.719960 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:01.719928 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcc5ce-667994994-qhjxm"] Nov 03 09:12:03.213393 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:03.213359 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" path="/var/lib/kubelet/pods/6d138326-bc46-4f60-9606-caa8d164b7cc/volumes" Nov 03 09:12:03.213895 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:03.213880 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b530646-50c4-4a45-9abe-35b786de0592" path="/var/lib/kubelet/pods/8b530646-50c4-4a45-9abe-35b786de0592/volumes" Nov 03 09:12:03.214271 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:03.214259 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe60606-a20a-4ce9-bdf9-acbec23c5b3a" path="/var/lib/kubelet/pods/cbe60606-a20a-4ce9-bdf9-acbec23c5b3a/volumes" Nov 03 09:12:06.577380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:06.577344 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4"] Nov 03 09:12:06.577941 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:06.577592 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" containerName="istio-proxy" containerID="cri-o://4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f" gracePeriod=30 Nov 03 09:12:06.628402 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:06.627794 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k"] Nov 03 09:12:06.628402 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:06.627995 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerName="istio-proxy" containerID="cri-o://85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389" gracePeriod=30 Nov 03 09:12:08.279623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.279583 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq"] Nov 03 09:12:08.280417 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280397 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" Nov 03 09:12:08.280417 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280417 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" Nov 03 09:12:08.280626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280448 5015 cpu_manager.go:414] "containerMap: removing container" podUID="cbe60606-a20a-4ce9-bdf9-acbec23c5b3a" containerName="maas-api" Nov 03 09:12:08.280626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280456 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe60606-a20a-4ce9-bdf9-acbec23c5b3a" containerName="maas-api" Nov 03 09:12:08.280626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280488 5015 cpu_manager.go:414] "containerMap: removing container" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" Nov 03 09:12:08.280626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280497 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" Nov 03 09:12:08.280626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280511 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8b530646-50c4-4a45-9abe-35b786de0592" containerName="istio-proxy" Nov 03 09:12:08.280626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280517 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b530646-50c4-4a45-9abe-35b786de0592" containerName="istio-proxy" Nov 03 09:12:08.280626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280534 5015 cpu_manager.go:414] "containerMap: removing container" podUID="9a24a04f-8d4f-4cbe-b131-78aa27d593c1" containerName="payload-processing" Nov 03 09:12:08.280626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280541 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a24a04f-8d4f-4cbe-b131-78aa27d593c1" containerName="payload-processing" Nov 03 09:12:08.280626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280585 5015 cpu_manager.go:414] "containerMap: removing container" podUID="c297a297-b068-459e-a32c-c56bdb9b317f" containerName="payload-pre-processing" Nov 03 09:12:08.280626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280593 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c297a297-b068-459e-a32c-c56bdb9b317f" containerName="payload-pre-processing" Nov 03 09:12:08.281095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280772 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="c297a297-b068-459e-a32c-c56bdb9b317f" containerName="payload-pre-processing" Nov 03 09:12:08.281095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280790 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d138326-bc46-4f60-9606-caa8d164b7cc" containerName="istio-proxy" Nov 03 09:12:08.281095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280804 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe60606-a20a-4ce9-bdf9-acbec23c5b3a" containerName="maas-api" Nov 03 09:12:08.281095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280814 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" Nov 03 09:12:08.281095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280821 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b530646-50c4-4a45-9abe-35b786de0592" containerName="istio-proxy" Nov 03 09:12:08.281095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280832 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a24a04f-8d4f-4cbe-b131-78aa27d593c1" containerName="payload-processing" Nov 03 09:12:08.281095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.280843 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" Nov 03 09:12:08.286174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.286149 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.338908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.338862 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9rlq\" (UniqueName: \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-kube-api-access-z9rlq\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.339467 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.339439 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-data\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.339745 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.339726 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2510f27d-c29c-43b0-bf06-43dc2d174f56-istiod-ca-cert\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.340004 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.339984 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-socket\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.351513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.350663 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-podinfo\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.351513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.350780 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-envoy\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.351513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.350813 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-certs\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.351513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.350909 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-token\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.351513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.350951 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-credential-socket\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.376104 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.376069 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq"] Nov 03 09:12:08.406813 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.397789 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-133b31-openshift-default-dockercfg-7smq5\"" Nov 03 09:12:08.452076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.452011 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-token\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.453202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.452660 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-credential-socket\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.453591 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.453178 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-credential-socket\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.454119 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.453468 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9rlq\" (UniqueName: \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-kube-api-access-z9rlq\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.454380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.454361 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-data\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.455135 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.455052 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-data\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.455283 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.455101 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2510f27d-c29c-43b0-bf06-43dc2d174f56-istiod-ca-cert\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.456381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.456363 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-socket\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.456871 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.456852 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-podinfo\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.457670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.457651 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-envoy\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.458226 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.458200 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-certs\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.458364 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.456765 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-socket\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.458456 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.456208 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2510f27d-c29c-43b0-bf06-43dc2d174f56-istiod-ca-cert\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.458713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.458693 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-certs\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.461527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.461504 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-podinfo\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.468264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.468210 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-envoy\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.512162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.512124 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-token\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.512540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.512519 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9rlq\" (UniqueName: \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-kube-api-access-z9rlq\") pod \"e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:08.601316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:08.601224 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:09.043022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:09.042984 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq"] Nov 03 09:12:09.043879 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:12:09.043853 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2510f27d_c29c_43b0_bf06_43dc2d174f56.slice/crio-8396abb0bc234bb09a2becfcc37d537b2bdfe4bf37e3fb22c6c412a03be2e00a WatchSource:0}: Error finding container 8396abb0bc234bb09a2becfcc37d537b2bdfe4bf37e3fb22c6c412a03be2e00a: Status 404 returned error can't find the container with id 8396abb0bc234bb09a2becfcc37d537b2bdfe4bf37e3fb22c6c412a03be2e00a Nov 03 09:12:09.046775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:09.046738 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:12:09.046885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:09.046815 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:12:09.046885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:09.046848 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:12:09.428781 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:09.428692 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" event={"ID":"2510f27d-c29c-43b0-bf06-43dc2d174f56","Type":"ContainerStarted","Data":"17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555"} Nov 03 09:12:09.428781 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:09.428759 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" event={"ID":"2510f27d-c29c-43b0-bf06-43dc2d174f56","Type":"ContainerStarted","Data":"8396abb0bc234bb09a2becfcc37d537b2bdfe4bf37e3fb22c6c412a03be2e00a"} Nov 03 09:12:09.550368 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:09.550305 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" podStartSLOduration=1.550286119 podStartE2EDuration="1.550286119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:12:09.547754599 +0000 UTC m=+1962.472696830" watchObservedRunningTime="2026-11-03 09:12:09.550286119 +0000 UTC m=+1962.475228340" Nov 03 09:12:09.601344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:09.601312 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:10.112199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:10.112158 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:12:10.112392 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:10.112238 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:12:10.536490 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:10.536434 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:12:10.536915 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:10.536521 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:12:10.608011 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:10.607977 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:11.462499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:11.461618 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:11.464879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:11.464856 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:12:11.936232 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:11.936200 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 09:12:11.966548 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:11.966526 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:12:12.013994 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.013909 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-istio-token\" (UniqueName: \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-istio-token\") pod \"0627db0c-9fec-4819-b096-ed00eff6f193\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " Nov 03 09:12:12.014641 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.014617 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-certs\") pod \"0627db0c-9fec-4819-b096-ed00eff6f193\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " Nov 03 09:12:12.014926 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.014903 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-certs" pod "0627db0c-9fec-4819-b096-ed00eff6f193" (UID: "0627db0c-9fec-4819-b096-ed00eff6f193"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:12.015045 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.015005 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-data\") pod \"0627db0c-9fec-4819-b096-ed00eff6f193\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " Nov 03 09:12:12.015304 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.015196 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-data" pod "0627db0c-9fec-4819-b096-ed00eff6f193" (UID: "0627db0c-9fec-4819-b096-ed00eff6f193"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:12.015304 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.015280 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0627db0c-9fec-4819-b096-ed00eff6f193-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0627db0c-9fec-4819-b096-ed00eff6f193-istiod-ca-cert\") pod \"0627db0c-9fec-4819-b096-ed00eff6f193\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " Nov 03 09:12:12.015447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.015332 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-envoy\") pod \"0627db0c-9fec-4819-b096-ed00eff6f193\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " Nov 03 09:12:12.015447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.015406 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-kube-api-access-cbw6s\" (UniqueName: \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-kube-api-access-cbw6s\") pod \"0627db0c-9fec-4819-b096-ed00eff6f193\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " Nov 03 09:12:12.016599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.015562 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-socket\") pod \"0627db0c-9fec-4819-b096-ed00eff6f193\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " Nov 03 09:12:12.016599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.015592 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-credential-socket\") pod \"0627db0c-9fec-4819-b096-ed00eff6f193\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " Nov 03 09:12:12.016599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.015710 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0627db0c-9fec-4819-b096-ed00eff6f193-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0627db0c-9fec-4819-b096-ed00eff6f193-istio-podinfo\") pod \"0627db0c-9fec-4819-b096-ed00eff6f193\" (UID: \"0627db0c-9fec-4819-b096-ed00eff6f193\") " Nov 03 09:12:12.016599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.015831 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0627db0c-9fec-4819-b096-ed00eff6f193-istiod-ca-cert" pod "0627db0c-9fec-4819-b096-ed00eff6f193" (UID: "0627db0c-9fec-4819-b096-ed00eff6f193"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:12:12.016599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.016248 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.016599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.016268 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.016599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.016281 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0627db0c-9fec-4819-b096-ed00eff6f193-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.016599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.016365 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-socket" pod "0627db0c-9fec-4819-b096-ed00eff6f193" (UID: "0627db0c-9fec-4819-b096-ed00eff6f193"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:12.017186 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.016662 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-credential-socket" pod "0627db0c-9fec-4819-b096-ed00eff6f193" (UID: "0627db0c-9fec-4819-b096-ed00eff6f193"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:12.017186 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.017164 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-istio-token" pod "0627db0c-9fec-4819-b096-ed00eff6f193" (UID: "0627db0c-9fec-4819-b096-ed00eff6f193"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:12.019439 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.019405 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0627db0c-9fec-4819-b096-ed00eff6f193-istio-podinfo" pod "0627db0c-9fec-4819-b096-ed00eff6f193" (UID: "0627db0c-9fec-4819-b096-ed00eff6f193"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:12:12.019883 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.019862 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-envoy" pod "0627db0c-9fec-4819-b096-ed00eff6f193" (UID: "0627db0c-9fec-4819-b096-ed00eff6f193"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:12.019964 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.019928 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-kube-api-access-cbw6s" pod "0627db0c-9fec-4819-b096-ed00eff6f193" (UID: "0627db0c-9fec-4819-b096-ed00eff6f193"). InnerVolumeSpecName "kube-api-access-cbw6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:12.117549 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.117516 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istiod-ca-cert\") pod \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " Nov 03 09:12:12.117728 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.117581 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-podinfo\") pod \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " Nov 03 09:12:12.117728 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.117628 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-token\" (UniqueName: \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-token\") pod \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " Nov 03 09:12:12.117805 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.117762 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-envoy\") pod \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " Nov 03 09:12:12.117805 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.117794 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-socket\") pod \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " Nov 03 09:12:12.117904 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.117850 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-certs\") pod \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " Nov 03 09:12:12.117904 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.117886 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-credential-socket\") pod \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " Nov 03 09:12:12.118025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.117952 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-kube-api-access-74mt6\" (UniqueName: \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-kube-api-access-74mt6\") pod \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " Nov 03 09:12:12.118025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.117964 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istiod-ca-cert" pod "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" (UID: "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:12:12.118129 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118034 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-data\") pod \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\" (UID: \"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c\") " Nov 03 09:12:12.118291 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118268 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-certs" pod "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" (UID: "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:12.118467 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118444 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-credential-socket" pod "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" (UID: "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:12.118467 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118455 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-data" pod "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" (UID: "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:12.118711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118615 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbw6s\" (UniqueName: \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-kube-api-access-cbw6s\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.118711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118632 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.118711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118650 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.118711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118669 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.118711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118670 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-socket" pod "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" (UID: "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:12.118711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118683 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.118711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118697 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0627db0c-9fec-4819-b096-ed00eff6f193-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.118711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118710 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0627db0c-9fec-4819-b096-ed00eff6f193-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.119008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118723 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.119008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118736 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.119008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.118748 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0627db0c-9fec-4819-b096-ed00eff6f193-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.120547 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.120447 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-kube-api-access-74mt6" pod "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" (UID: "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c"). InnerVolumeSpecName "kube-api-access-74mt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:12.120655 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.120593 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-podinfo" pod "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" (UID: "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:12:12.120763 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.120738 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-envoy" pod "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" (UID: "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:12.120802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.120742 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-token" pod "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" (UID: "0d1513d8-3e7c-4733-aa56-9ccd7eeac09c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:12.228693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.228658 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.228693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.228687 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.228693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.228697 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.229038 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.228706 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.229038 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.228716 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74mt6\" (UniqueName: \"kubernetes.io/projected/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c-kube-api-access-74mt6\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:12.472171 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.472119 5015 generic.go:356] "Generic (PLEG): container finished" podID="0627db0c-9fec-4819-b096-ed00eff6f193" containerID="4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f" exitCode=0 Nov 03 09:12:12.472380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.472249 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" Nov 03 09:12:12.472380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.472297 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" event={"ID":"0627db0c-9fec-4819-b096-ed00eff6f193","Type":"ContainerDied","Data":"4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f"} Nov 03 09:12:12.472380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.472340 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4" event={"ID":"0627db0c-9fec-4819-b096-ed00eff6f193","Type":"ContainerDied","Data":"3a9f68049451ed400bd447bfd5d3979459dcb87e2e6797b18fa31e5f74b0f951"} Nov 03 09:12:12.472380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.472362 5015 scope.go:122] "RemoveContainer" containerID="4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f" Nov 03 09:12:12.474735 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.474679 5015 generic.go:356] "Generic (PLEG): container finished" podID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerID="85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389" exitCode=0 Nov 03 09:12:12.474860 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.474726 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" Nov 03 09:12:12.475686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.475662 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" event={"ID":"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c","Type":"ContainerDied","Data":"85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389"} Nov 03 09:12:12.475814 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.475710 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k" event={"ID":"0d1513d8-3e7c-4733-aa56-9ccd7eeac09c","Type":"ContainerDied","Data":"2f8079178d1e91b64cb2a053480f10c8aa48e26fa8808418246770211683ae06"} Nov 03 09:12:12.489160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.489141 5015 scope.go:122] "RemoveContainer" containerID="4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f" Nov 03 09:12:12.489591 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:12:12.489565 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f\": container with ID starting with 4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f not found: ID does not exist" containerID="4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f" Nov 03 09:12:12.489678 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.489603 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f"} err="failed to get container status \"4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f\": rpc error: code = NotFound desc = could not find container \"4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f\": container with ID starting with 4b148902d64e94fe5a7f6cc789b5ac889f11cef32991a0eccc875ce8283df16f not found: ID does not exist" Nov 03 09:12:12.489678 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.489628 5015 scope.go:122] "RemoveContainer" containerID="85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389" Nov 03 09:12:12.505268 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.505236 5015 scope.go:122] "RemoveContainer" containerID="85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389" Nov 03 09:12:12.505679 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:12:12.505649 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389\": container with ID starting with 85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389 not found: ID does not exist" containerID="85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389" Nov 03 09:12:12.505782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.505690 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389"} err="failed to get container status \"85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389\": rpc error: code = NotFound desc = could not find container \"85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389\": container with ID starting with 85b4a582a3de32c4364f2c96df2901d758f51c34651e8a9c0040a851017f0389 not found: ID does not exist" Nov 03 09:12:12.665425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.665376 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4"] Nov 03 09:12:12.670122 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.670087 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-gktx4"] Nov 03 09:12:12.785352 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.785261 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k"] Nov 03 09:12:12.795886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.795847 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-tqv2k"] Nov 03 09:12:12.817941 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.817910 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn"] Nov 03 09:12:12.818533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.818518 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerName="istio-proxy" Nov 03 09:12:12.818575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.818534 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerName="istio-proxy" Nov 03 09:12:12.818575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.818555 5015 cpu_manager.go:414] "containerMap: removing container" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" Nov 03 09:12:12.818575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.818561 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911aeeac-4f94-4ee1-a6c6-46db527d4f0f" containerName="istio-proxy" Nov 03 09:12:12.818575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.818568 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" containerName="istio-proxy" Nov 03 09:12:12.818575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.818573 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" containerName="istio-proxy" Nov 03 09:12:12.818747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.818656 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" containerName="istio-proxy" Nov 03 09:12:12.818747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.818668 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" containerName="istio-proxy" Nov 03 09:12:12.823873 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.823855 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:12.878306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.878278 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-04934e\"/\"kube-root-ca.crt\"" Nov 03 09:12:12.878543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.878358 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-04934e\"/\"openshift-service-ca.crt\"" Nov 03 09:12:12.878764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.878742 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-04934e\"/\"ipp-route-04934e-kserve-self-signed-certs\"" Nov 03 09:12:12.878764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.878746 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-04934e\"/\"default-dockercfg-lnkdn\"" Nov 03 09:12:12.880380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.880355 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn"] Nov 03 09:12:12.947894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.947855 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-dshm\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:12.948359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.947912 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-home\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:12.948359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.948114 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-model-cache\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:12.948359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.948144 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgxgh\" (UniqueName: \"kubernetes.io/projected/b72ebb8a-18db-4212-8513-59185daa0074-kube-api-access-bgxgh\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:12.948359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.948273 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-tmp-dir\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:12.948359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:12.948329 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72ebb8a-18db-4212-8513-59185daa0074-tls-certs\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.050124 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.050032 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-model-cache\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.050124 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.050078 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgxgh\" (UniqueName: \"kubernetes.io/projected/b72ebb8a-18db-4212-8513-59185daa0074-kube-api-access-bgxgh\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.050382 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.050136 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-tmp-dir\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.050382 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.050163 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72ebb8a-18db-4212-8513-59185daa0074-tls-certs\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.050587 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.050556 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-dshm\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.050809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.050779 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-home\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.050924 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.050585 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-model-cache\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.051040 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.050610 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-tmp-dir\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.051284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.051260 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-home\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.053422 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.053395 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-dshm\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.053744 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.053724 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72ebb8a-18db-4212-8513-59185daa0074-tls-certs\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.110296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.110255 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgxgh\" (UniqueName: \"kubernetes.io/projected/b72ebb8a-18db-4212-8513-59185daa0074-kube-api-access-bgxgh\") pod \"ipp-route-04934e-kserve-7dd67bb465-4qxjn\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.136926 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.136894 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:13.211974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.211939 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0627db0c-9fec-4819-b096-ed00eff6f193" path="/var/lib/kubelet/pods/0627db0c-9fec-4819-b096-ed00eff6f193/volumes" Nov 03 09:12:13.213440 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.213416 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d1513d8-3e7c-4733-aa56-9ccd7eeac09c" path="/var/lib/kubelet/pods/0d1513d8-3e7c-4733-aa56-9ccd7eeac09c/volumes" Nov 03 09:12:13.519519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.519490 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn"] Nov 03 09:12:13.520175 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:12:13.520147 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb72ebb8a_18db_4212_8513_59185daa0074.slice/crio-bac7190cd09ab19a3d8770dd4a8408619c120f9c7ddb56a2ac4c351998e13e8c WatchSource:0}: Error finding container bac7190cd09ab19a3d8770dd4a8408619c120f9c7ddb56a2ac4c351998e13e8c: Status 404 returned error can't find the container with id bac7190cd09ab19a3d8770dd4a8408619c120f9c7ddb56a2ac4c351998e13e8c Nov 03 09:12:13.716264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.716216 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs"] Nov 03 09:12:13.720797 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.720771 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:12:13.776603 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.776515 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c32d41\"" Nov 03 09:12:13.776762 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.776671 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c32d41-dockercfg-l4rvg\"" Nov 03 09:12:13.781404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.781376 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs"] Nov 03 09:12:13.869423 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.869386 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lrrz\" (UniqueName: \"kubernetes.io/projected/a758c014-84c7-425e-b9cb-5ea42c09625d-kube-api-access-8lrrz\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs\" (UID: \"a758c014-84c7-425e-b9cb-5ea42c09625d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:12:13.869675 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.869428 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a758c014-84c7-425e-b9cb-5ea42c09625d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs\" (UID: \"a758c014-84c7-425e-b9cb-5ea42c09625d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:12:13.971249 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.971209 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lrrz\" (UniqueName: \"kubernetes.io/projected/a758c014-84c7-425e-b9cb-5ea42c09625d-kube-api-access-8lrrz\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs\" (UID: \"a758c014-84c7-425e-b9cb-5ea42c09625d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:12:13.971745 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.971281 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a758c014-84c7-425e-b9cb-5ea42c09625d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs\" (UID: \"a758c014-84c7-425e-b9cb-5ea42c09625d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:12:13.972149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:13.972124 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a758c014-84c7-425e-b9cb-5ea42c09625d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs\" (UID: \"a758c014-84c7-425e-b9cb-5ea42c09625d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:12:14.034857 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.034782 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lrrz\" (UniqueName: \"kubernetes.io/projected/a758c014-84c7-425e-b9cb-5ea42c09625d-kube-api-access-8lrrz\") pod \"payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs\" (UID: \"a758c014-84c7-425e-b9cb-5ea42c09625d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:12:14.039354 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.039325 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:12:14.422884 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.422854 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs"] Nov 03 09:12:14.424446 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:12:14.424406 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda758c014_84c7_425e_b9cb_5ea42c09625d.slice/crio-1bdaefd6d9ef31707ec47d83557892c995f1994257da481f91639b67be68f6f3 WatchSource:0}: Error finding container 1bdaefd6d9ef31707ec47d83557892c995f1994257da481f91639b67be68f6f3: Status 404 returned error can't find the container with id 1bdaefd6d9ef31707ec47d83557892c995f1994257da481f91639b67be68f6f3 Nov 03 09:12:14.426825 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.426802 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68"] Nov 03 09:12:14.432769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.432745 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:12:14.496961 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.496928 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" event={"ID":"b72ebb8a-18db-4212-8513-59185daa0074","Type":"ContainerStarted","Data":"e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc"} Nov 03 09:12:14.497147 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.496976 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" event={"ID":"b72ebb8a-18db-4212-8513-59185daa0074","Type":"ContainerStarted","Data":"bac7190cd09ab19a3d8770dd4a8408619c120f9c7ddb56a2ac4c351998e13e8c"} Nov 03 09:12:14.498433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.498409 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" event={"ID":"a758c014-84c7-425e-b9cb-5ea42c09625d","Type":"ContainerStarted","Data":"1bdaefd6d9ef31707ec47d83557892c995f1994257da481f91639b67be68f6f3"} Nov 03 09:12:14.528120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.528093 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68"] Nov 03 09:12:14.581221 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.581184 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62bcg\" (UniqueName: \"kubernetes.io/projected/56530cb6-7d91-469a-96cb-4049f1ec5e5a-kube-api-access-62bcg\") pod \"payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68\" (UID: \"56530cb6-7d91-469a-96cb-4049f1ec5e5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:12:14.581433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.581400 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56530cb6-7d91-469a-96cb-4049f1ec5e5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68\" (UID: \"56530cb6-7d91-469a-96cb-4049f1ec5e5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:12:14.667220 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.667159 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" podStartSLOduration=2.667144135 podStartE2EDuration="2.667144135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:12:14.66563219 +0000 UTC m=+1967.590574464" watchObservedRunningTime="2026-11-03 09:12:14.667144135 +0000 UTC m=+1967.592086350" Nov 03 09:12:14.683092 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.682999 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62bcg\" (UniqueName: \"kubernetes.io/projected/56530cb6-7d91-469a-96cb-4049f1ec5e5a-kube-api-access-62bcg\") pod \"payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68\" (UID: \"56530cb6-7d91-469a-96cb-4049f1ec5e5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:12:14.683233 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.683186 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56530cb6-7d91-469a-96cb-4049f1ec5e5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68\" (UID: \"56530cb6-7d91-469a-96cb-4049f1ec5e5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:12:14.683831 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.683810 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56530cb6-7d91-469a-96cb-4049f1ec5e5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68\" (UID: \"56530cb6-7d91-469a-96cb-4049f1ec5e5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:12:14.745308 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.745276 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62bcg\" (UniqueName: \"kubernetes.io/projected/56530cb6-7d91-469a-96cb-4049f1ec5e5a-kube-api-access-62bcg\") pod \"payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68\" (UID: \"56530cb6-7d91-469a-96cb-4049f1ec5e5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:12:14.753768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:14.753740 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:12:15.137582 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:15.137545 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:15.168699 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:12:15.167710 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56530cb6_7d91_469a_96cb_4049f1ec5e5a.slice/crio-02346db149fec693a74b92222d80781f5ae67e52a1476da93506ee6ee5636489 WatchSource:0}: Error finding container 02346db149fec693a74b92222d80781f5ae67e52a1476da93506ee6ee5636489: Status 404 returned error can't find the container with id 02346db149fec693a74b92222d80781f5ae67e52a1476da93506ee6ee5636489 Nov 03 09:12:15.168699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:15.168307 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68"] Nov 03 09:12:15.509034 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:15.509001 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" event={"ID":"a758c014-84c7-425e-b9cb-5ea42c09625d","Type":"ContainerStarted","Data":"229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd"} Nov 03 09:12:15.509254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:15.509209 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:12:15.510953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:15.510930 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" event={"ID":"56530cb6-7d91-469a-96cb-4049f1ec5e5a","Type":"ContainerStarted","Data":"f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517"} Nov 03 09:12:15.511041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:15.510976 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" event={"ID":"56530cb6-7d91-469a-96cb-4049f1ec5e5a","Type":"ContainerStarted","Data":"02346db149fec693a74b92222d80781f5ae67e52a1476da93506ee6ee5636489"} Nov 03 09:12:15.511041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:15.511010 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:12:15.630749 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:15.630695 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" podStartSLOduration=2.6306791670000003 podStartE2EDuration="2.630679167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:12:15.627516519 +0000 UTC m=+1968.552458730" watchObservedRunningTime="2026-11-03 09:12:15.630679167 +0000 UTC m=+1968.555621405" Nov 03 09:12:15.747298 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:15.747227 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" podStartSLOduration=1.747213975 podStartE2EDuration="1.747213975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:12:15.746212977 +0000 UTC m=+1968.671155220" watchObservedRunningTime="2026-11-03 09:12:15.747213975 +0000 UTC m=+1968.672156192" Nov 03 09:12:16.019866 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.019826 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77"] Nov 03 09:12:16.024240 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.024217 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:12:16.079895 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.079856 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-133b31\"" Nov 03 09:12:16.082908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.082880 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77"] Nov 03 09:12:16.107653 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.107602 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f67346f-7c34-460d-898e-30dd07e22f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77\" (UID: \"3f67346f-7c34-460d-898e-30dd07e22f12\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:12:16.107842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.107812 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxx6\" (UniqueName: \"kubernetes.io/projected/3f67346f-7c34-460d-898e-30dd07e22f12-kube-api-access-rsxx6\") pod \"payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77\" (UID: \"3f67346f-7c34-460d-898e-30dd07e22f12\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:12:16.143498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.141803 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-133b31-dockercfg-vsk5t\"" Nov 03 09:12:16.209065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.209016 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f67346f-7c34-460d-898e-30dd07e22f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77\" (UID: \"3f67346f-7c34-460d-898e-30dd07e22f12\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:12:16.209288 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.209249 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxx6\" (UniqueName: \"kubernetes.io/projected/3f67346f-7c34-460d-898e-30dd07e22f12-kube-api-access-rsxx6\") pod \"payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77\" (UID: \"3f67346f-7c34-460d-898e-30dd07e22f12\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:12:16.210990 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.210963 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f67346f-7c34-460d-898e-30dd07e22f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77\" (UID: \"3f67346f-7c34-460d-898e-30dd07e22f12\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:12:16.215761 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.215734 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7"] Nov 03 09:12:16.221539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.221515 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:12:16.231324 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.231282 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7"] Nov 03 09:12:16.270664 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.270431 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxx6\" (UniqueName: \"kubernetes.io/projected/3f67346f-7c34-460d-898e-30dd07e22f12-kube-api-access-rsxx6\") pod \"payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77\" (UID: \"3f67346f-7c34-460d-898e-30dd07e22f12\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:12:16.310250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.310215 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjmpt\" (UniqueName: \"kubernetes.io/projected/3449b460-2c34-428a-a2c7-71b5dba3b16a-kube-api-access-rjmpt\") pod \"payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7\" (UID: \"3449b460-2c34-428a-a2c7-71b5dba3b16a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:12:16.310250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.310254 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3449b460-2c34-428a-a2c7-71b5dba3b16a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7\" (UID: \"3449b460-2c34-428a-a2c7-71b5dba3b16a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:12:16.341655 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.341605 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:12:16.411538 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.411448 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjmpt\" (UniqueName: \"kubernetes.io/projected/3449b460-2c34-428a-a2c7-71b5dba3b16a-kube-api-access-rjmpt\") pod \"payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7\" (UID: \"3449b460-2c34-428a-a2c7-71b5dba3b16a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:12:16.411538 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.411512 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3449b460-2c34-428a-a2c7-71b5dba3b16a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7\" (UID: \"3449b460-2c34-428a-a2c7-71b5dba3b16a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:12:16.412801 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.412563 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3449b460-2c34-428a-a2c7-71b5dba3b16a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7\" (UID: \"3449b460-2c34-428a-a2c7-71b5dba3b16a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:12:16.479641 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.479603 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjmpt\" (UniqueName: \"kubernetes.io/projected/3449b460-2c34-428a-a2c7-71b5dba3b16a-kube-api-access-rjmpt\") pod \"payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7\" (UID: \"3449b460-2c34-428a-a2c7-71b5dba3b16a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:12:16.541783 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.541690 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:12:16.746120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.746078 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77"] Nov 03 09:12:16.747523 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:12:16.747498 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f67346f_7c34_460d_898e_30dd07e22f12.slice/crio-228d02b36efd7dc3b21a1009cd63d681080b04254fc00ed3bbc4527dae11181a WatchSource:0}: Error finding container 228d02b36efd7dc3b21a1009cd63d681080b04254fc00ed3bbc4527dae11181a: Status 404 returned error can't find the container with id 228d02b36efd7dc3b21a1009cd63d681080b04254fc00ed3bbc4527dae11181a Nov 03 09:12:16.905593 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:12:16.905553 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:12:16.928893 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:16.928864 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7"] Nov 03 09:12:16.930537 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:12:16.930508 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3449b460_2c34_428a_a2c7_71b5dba3b16a.slice/crio-fa93cd7b4cfc2e79d70b0f3603d2a45fb1adc817ff20494ef1e3929005f9ab26 WatchSource:0}: Error finding container fa93cd7b4cfc2e79d70b0f3603d2a45fb1adc817ff20494ef1e3929005f9ab26: Status 404 returned error can't find the container with id fa93cd7b4cfc2e79d70b0f3603d2a45fb1adc817ff20494ef1e3929005f9ab26 Nov 03 09:12:17.531233 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:17.531180 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" event={"ID":"3449b460-2c34-428a-a2c7-71b5dba3b16a","Type":"ContainerStarted","Data":"5dfbb7ceea0a66fd725f1fa82c31dfa3e2a2944d4cc6468bd26802472dc29065"} Nov 03 09:12:17.531791 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:17.531245 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" event={"ID":"3449b460-2c34-428a-a2c7-71b5dba3b16a","Type":"ContainerStarted","Data":"fa93cd7b4cfc2e79d70b0f3603d2a45fb1adc817ff20494ef1e3929005f9ab26"} Nov 03 09:12:17.531791 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:17.531272 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:12:17.533205 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:17.533176 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" event={"ID":"3f67346f-7c34-460d-898e-30dd07e22f12","Type":"ContainerStarted","Data":"139c81dbe0a097974500d993503c57e89ac7cebd67b521c89122ff28c1040c4b"} Nov 03 09:12:17.533351 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:17.533227 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" event={"ID":"3f67346f-7c34-460d-898e-30dd07e22f12","Type":"ContainerStarted","Data":"228d02b36efd7dc3b21a1009cd63d681080b04254fc00ed3bbc4527dae11181a"} Nov 03 09:12:17.533351 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:17.533268 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:12:17.651035 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:17.650965 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" podStartSLOduration=1.650942504 podStartE2EDuration="1.650942504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:12:17.646961217 +0000 UTC m=+1970.571903434" watchObservedRunningTime="2026-11-03 09:12:17.650942504 +0000 UTC m=+1970.575884722" Nov 03 09:12:17.775501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:17.772930 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" podStartSLOduration=2.772905727 podStartE2EDuration="2.772905727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:12:17.766634239 +0000 UTC m=+1970.691576479" watchObservedRunningTime="2026-11-03 09:12:17.772905727 +0000 UTC m=+1970.697847943" Nov 03 09:12:23.137972 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:23.137932 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:23.156050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:23.156023 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:23.602942 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:23.602914 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:12:25.612508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:25.612374 5015 generic.go:356] "Generic (PLEG): container finished" podID="56530cb6-7d91-469a-96cb-4049f1ec5e5a" containerID="f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517" exitCode=1 Nov 03 09:12:25.612508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:25.612448 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" event={"ID":"56530cb6-7d91-469a-96cb-4049f1ec5e5a","Type":"ContainerDied","Data":"f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517"} Nov 03 09:12:25.613044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:25.612897 5015 scope.go:122] "RemoveContainer" containerID="f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517" Nov 03 09:12:26.523280 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:26.523250 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:12:26.624360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:26.624328 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" event={"ID":"56530cb6-7d91-469a-96cb-4049f1ec5e5a","Type":"ContainerStarted","Data":"44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec"} Nov 03 09:12:26.624790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:26.624539 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:12:28.544306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:28.544265 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:12:28.544712 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:28.544695 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:12:33.012971 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:33.012938 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt"] Nov 03 09:12:33.136738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:33.136690 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt"] Nov 03 09:12:33.933562 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:33.933523 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw"] Nov 03 09:12:33.939858 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:33.939827 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:33.995746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:33.995708 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-121ee6-openshift-default-dockercfg-dcnsq\"" Nov 03 09:12:33.999820 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:33.999789 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw"] Nov 03 09:12:34.050354 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.050317 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-certs\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.050786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.050386 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-token\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.050786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.050506 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istiod-ca-cert\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.050786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.050630 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-socket\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.050786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.050694 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khg2z\" (UniqueName: \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-kube-api-access-khg2z\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.050786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.050752 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-credential-socket\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.050786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.050783 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-podinfo\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.050997 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.050820 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-data\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.050997 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.050845 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-envoy\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.152291 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.152258 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istiod-ca-cert\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.152517 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.152364 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-socket\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.152517 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.152454 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khg2z\" (UniqueName: \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-kube-api-access-khg2z\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.152679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.152525 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-credential-socket\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.152679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.152567 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-podinfo\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.152679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.152597 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-data\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.152679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.152627 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-envoy\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.152679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.152658 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-certs\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.152922 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.152736 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-token\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.152922 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.152849 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-socket\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.153055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.153024 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-credential-socket\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.153195 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.153168 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istiod-ca-cert\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.153195 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.153186 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-data\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.153364 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.153310 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-certs\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.155886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.155862 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-envoy\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.156105 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.156087 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-podinfo\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.239132 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.239065 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-token\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.239325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.239269 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khg2z\" (UniqueName: \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-kube-api-access-khg2z\") pod \"e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.257012 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.256739 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:34.711149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.711112 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw"] Nov 03 09:12:34.711424 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:12:34.711402 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc56a0ca_ca23_4ef2_a13b_3368c3944e8b.slice/crio-5428c13964cde99a61e05003182a3192114baec7203f8af417eb57d068d2e561 WatchSource:0}: Error finding container 5428c13964cde99a61e05003182a3192114baec7203f8af417eb57d068d2e561: Status 404 returned error can't find the container with id 5428c13964cde99a61e05003182a3192114baec7203f8af417eb57d068d2e561 Nov 03 09:12:34.714433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.714397 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:12:34.714539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.714465 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:12:34.714584 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:34.714540 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:12:35.690185 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:35.690150 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" event={"ID":"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b","Type":"ContainerStarted","Data":"a84a08297e12eea2f078d0c1e2a2f8b206b6709c2d03c8db057130c1d05e00a4"} Nov 03 09:12:35.690610 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:35.690206 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" event={"ID":"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b","Type":"ContainerStarted","Data":"5428c13964cde99a61e05003182a3192114baec7203f8af417eb57d068d2e561"} Nov 03 09:12:35.909677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:35.909625 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" podStartSLOduration=2.90960607 podStartE2EDuration="2.90960607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:12:35.859343161 +0000 UTC m=+1988.784285379" watchObservedRunningTime="2026-11-03 09:12:35.90960607 +0000 UTC m=+1988.834548287" Nov 03 09:12:36.257720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:36.257679 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:36.263550 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:36.263523 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:36.698854 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:36.698773 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:36.699842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:36.699820 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:12:37.634666 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:37.634631 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:12:38.013711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.013666 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" podUID="d6833a04-dae7-4186-aa13-79f7db830a2d" containerName="payload-processing" containerID="cri-o://39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97" gracePeriod=25 Nov 03 09:12:38.137762 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.137717 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" podUID="c1f0e49a-ace8-41e6-941d-5486094e76f8" containerName="payload-pre-processing" containerID="cri-o://b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494" gracePeriod=25 Nov 03 09:12:38.291489 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.291448 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:12:38.420874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.420845 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6833a04-dae7-4186-aa13-79f7db830a2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6833a04-dae7-4186-aa13-79f7db830a2d-plugins-config-volume\") pod \"d6833a04-dae7-4186-aa13-79f7db830a2d\" (UID: \"d6833a04-dae7-4186-aa13-79f7db830a2d\") " Nov 03 09:12:38.421061 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.420920 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6833a04-dae7-4186-aa13-79f7db830a2d-kube-api-access-h8x72\" (UniqueName: \"kubernetes.io/projected/d6833a04-dae7-4186-aa13-79f7db830a2d-kube-api-access-h8x72\") pod \"d6833a04-dae7-4186-aa13-79f7db830a2d\" (UID: \"d6833a04-dae7-4186-aa13-79f7db830a2d\") " Nov 03 09:12:38.421309 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.421275 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6833a04-dae7-4186-aa13-79f7db830a2d-plugins-config-volume" pod "d6833a04-dae7-4186-aa13-79f7db830a2d" (UID: "d6833a04-dae7-4186-aa13-79f7db830a2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:12:38.421653 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.421627 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6833a04-dae7-4186-aa13-79f7db830a2d-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:38.424304 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.424272 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6833a04-dae7-4186-aa13-79f7db830a2d-kube-api-access-h8x72" pod "d6833a04-dae7-4186-aa13-79f7db830a2d" (UID: "d6833a04-dae7-4186-aa13-79f7db830a2d"). InnerVolumeSpecName "kube-api-access-h8x72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:38.434004 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.433965 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:12:38.523260 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.523173 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8x72\" (UniqueName: \"kubernetes.io/projected/d6833a04-dae7-4186-aa13-79f7db830a2d-kube-api-access-h8x72\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:38.624140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.624100 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f0e49a-ace8-41e6-941d-5486094e76f8-kube-api-access-fswvt\" (UniqueName: \"kubernetes.io/projected/c1f0e49a-ace8-41e6-941d-5486094e76f8-kube-api-access-fswvt\") pod \"c1f0e49a-ace8-41e6-941d-5486094e76f8\" (UID: \"c1f0e49a-ace8-41e6-941d-5486094e76f8\") " Nov 03 09:12:38.624303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.624234 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f0e49a-ace8-41e6-941d-5486094e76f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f0e49a-ace8-41e6-941d-5486094e76f8-plugins-config-volume\") pod \"c1f0e49a-ace8-41e6-941d-5486094e76f8\" (UID: \"c1f0e49a-ace8-41e6-941d-5486094e76f8\") " Nov 03 09:12:38.624669 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.624640 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f0e49a-ace8-41e6-941d-5486094e76f8-plugins-config-volume" pod "c1f0e49a-ace8-41e6-941d-5486094e76f8" (UID: "c1f0e49a-ace8-41e6-941d-5486094e76f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:12:38.624896 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.624877 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f0e49a-ace8-41e6-941d-5486094e76f8-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:38.626586 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.626565 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f0e49a-ace8-41e6-941d-5486094e76f8-kube-api-access-fswvt" pod "c1f0e49a-ace8-41e6-941d-5486094e76f8" (UID: "c1f0e49a-ace8-41e6-941d-5486094e76f8"). InnerVolumeSpecName "kube-api-access-fswvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:38.714557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.714520 5015 generic.go:356] "Generic (PLEG): container finished" podID="d6833a04-dae7-4186-aa13-79f7db830a2d" containerID="39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97" exitCode=0 Nov 03 09:12:38.714738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.714573 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" Nov 03 09:12:38.714738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.714621 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" event={"ID":"d6833a04-dae7-4186-aa13-79f7db830a2d","Type":"ContainerDied","Data":"39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97"} Nov 03 09:12:38.714738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.714675 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt" event={"ID":"d6833a04-dae7-4186-aa13-79f7db830a2d","Type":"ContainerDied","Data":"f191c03879b9bfd3edbc66341b26ae57673509fc259cd4a218e32536e38ab252"} Nov 03 09:12:38.714738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.714702 5015 scope.go:122] "RemoveContainer" containerID="39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97" Nov 03 09:12:38.716131 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.716094 5015 generic.go:356] "Generic (PLEG): container finished" podID="c1f0e49a-ace8-41e6-941d-5486094e76f8" containerID="b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494" exitCode=0 Nov 03 09:12:38.716238 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.716150 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" Nov 03 09:12:38.716238 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.716182 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" event={"ID":"c1f0e49a-ace8-41e6-941d-5486094e76f8","Type":"ContainerDied","Data":"b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494"} Nov 03 09:12:38.716238 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.716235 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt" event={"ID":"c1f0e49a-ace8-41e6-941d-5486094e76f8","Type":"ContainerDied","Data":"288da991d5d80ef55408e04fbcf3eb99bb1357566455952ae7ad41f4cc288706"} Nov 03 09:12:38.725778 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.725752 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fswvt\" (UniqueName: \"kubernetes.io/projected/c1f0e49a-ace8-41e6-941d-5486094e76f8-kube-api-access-fswvt\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:38.727956 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.727939 5015 scope.go:122] "RemoveContainer" containerID="39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97" Nov 03 09:12:38.728308 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:12:38.728289 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97\": container with ID starting with 39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97 not found: ID does not exist" containerID="39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97" Nov 03 09:12:38.728382 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.728320 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97"} err="failed to get container status \"39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97\": rpc error: code = NotFound desc = could not find container \"39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97\": container with ID starting with 39ae283d50e7e9f85519c5f9460d31576e6992cc3651cd688bba9ac3f9518b97 not found: ID does not exist" Nov 03 09:12:38.728382 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.728347 5015 scope.go:122] "RemoveContainer" containerID="b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494" Nov 03 09:12:38.740459 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.740433 5015 scope.go:122] "RemoveContainer" containerID="b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494" Nov 03 09:12:38.740895 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:12:38.740855 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494\": container with ID starting with b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494 not found: ID does not exist" containerID="b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494" Nov 03 09:12:38.741006 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.740906 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494"} err="failed to get container status \"b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494\": rpc error: code = NotFound desc = could not find container \"b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494\": container with ID starting with b2c4bac2a9b70df77f7e334984c1a87d4e1cfefb955d8a8020564b13e4827494 not found: ID does not exist" Nov 03 09:12:38.885120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.885024 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt"] Nov 03 09:12:38.893215 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:38.893185 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-57976f5577-r5mlt"] Nov 03 09:12:39.010826 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:39.010793 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt"] Nov 03 09:12:39.016054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:39.016022 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7b956bc895-8l2pt"] Nov 03 09:12:39.206023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:39.205944 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f0e49a-ace8-41e6-941d-5486094e76f8" path="/var/lib/kubelet/pods/c1f0e49a-ace8-41e6-941d-5486094e76f8/volumes" Nov 03 09:12:39.206404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:39.206388 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6833a04-dae7-4186-aa13-79f7db830a2d" path="/var/lib/kubelet/pods/d6833a04-dae7-4186-aa13-79f7db830a2d/volumes" Nov 03 09:12:42.442321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.442260 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx"] Nov 03 09:12:42.444011 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.443981 5015 cpu_manager.go:414] "containerMap: removing container" podUID="c1f0e49a-ace8-41e6-941d-5486094e76f8" containerName="payload-pre-processing" Nov 03 09:12:42.444284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.444266 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f0e49a-ace8-41e6-941d-5486094e76f8" containerName="payload-pre-processing" Nov 03 09:12:42.444470 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.444419 5015 cpu_manager.go:414] "containerMap: removing container" podUID="d6833a04-dae7-4186-aa13-79f7db830a2d" containerName="payload-processing" Nov 03 09:12:42.444470 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.444436 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6833a04-dae7-4186-aa13-79f7db830a2d" containerName="payload-processing" Nov 03 09:12:42.444753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.444660 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f0e49a-ace8-41e6-941d-5486094e76f8" containerName="payload-pre-processing" Nov 03 09:12:42.444753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.444683 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6833a04-dae7-4186-aa13-79f7db830a2d" containerName="payload-processing" Nov 03 09:12:42.450981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.450949 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:12:42.480542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.480509 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq9pw\" (UniqueName: \"kubernetes.io/projected/28e9f4e5-1c70-40c9-a56d-dff029b79b11-kube-api-access-kq9pw\") pod \"payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx\" (UID: \"28e9f4e5-1c70-40c9-a56d-dff029b79b11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:12:42.480962 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.480938 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e9f4e5-1c70-40c9-a56d-dff029b79b11-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx\" (UID: \"28e9f4e5-1c70-40c9-a56d-dff029b79b11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:12:42.496981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.496927 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx"] Nov 03 09:12:42.540084 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.540053 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-121ee6\"" Nov 03 09:12:42.540239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.540202 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-121ee6-dockercfg-bztsl\"" Nov 03 09:12:42.582883 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.582823 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e9f4e5-1c70-40c9-a56d-dff029b79b11-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx\" (UID: \"28e9f4e5-1c70-40c9-a56d-dff029b79b11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:12:42.583064 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.582907 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9pw\" (UniqueName: \"kubernetes.io/projected/28e9f4e5-1c70-40c9-a56d-dff029b79b11-kube-api-access-kq9pw\") pod \"payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx\" (UID: \"28e9f4e5-1c70-40c9-a56d-dff029b79b11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:12:42.583683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.583663 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e9f4e5-1c70-40c9-a56d-dff029b79b11-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx\" (UID: \"28e9f4e5-1c70-40c9-a56d-dff029b79b11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:12:42.594823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.594786 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7"] Nov 03 09:12:42.599533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.599509 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:12:42.642798 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.642764 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9pw\" (UniqueName: \"kubernetes.io/projected/28e9f4e5-1c70-40c9-a56d-dff029b79b11-kube-api-access-kq9pw\") pod \"payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx\" (UID: \"28e9f4e5-1c70-40c9-a56d-dff029b79b11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:12:42.649910 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.649875 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7"] Nov 03 09:12:42.684686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.684642 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad07039-b190-421f-8c5f-32be14742a24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7\" (UID: \"0ad07039-b190-421f-8c5f-32be14742a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:12:42.684877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.684834 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pgz\" (UniqueName: \"kubernetes.io/projected/0ad07039-b190-421f-8c5f-32be14742a24-kube-api-access-c9pgz\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7\" (UID: \"0ad07039-b190-421f-8c5f-32be14742a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:12:42.781146 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.781106 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:12:42.786265 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.786237 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad07039-b190-421f-8c5f-32be14742a24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7\" (UID: \"0ad07039-b190-421f-8c5f-32be14742a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:12:42.786372 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.786350 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pgz\" (UniqueName: \"kubernetes.io/projected/0ad07039-b190-421f-8c5f-32be14742a24-kube-api-access-c9pgz\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7\" (UID: \"0ad07039-b190-421f-8c5f-32be14742a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:12:42.786974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.786956 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad07039-b190-421f-8c5f-32be14742a24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7\" (UID: \"0ad07039-b190-421f-8c5f-32be14742a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:12:42.846679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.846585 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pgz\" (UniqueName: \"kubernetes.io/projected/0ad07039-b190-421f-8c5f-32be14742a24-kube-api-access-c9pgz\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7\" (UID: \"0ad07039-b190-421f-8c5f-32be14742a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:12:42.914748 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:42.914690 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:12:43.166613 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:43.166582 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx"] Nov 03 09:12:43.167019 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:12:43.166978 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e9f4e5_1c70_40c9_a56d_dff029b79b11.slice/crio-9ec423fb0b8cbc0be567cbff0bae37f5227a788dfc3af0fc6b7e9cd93134462d WatchSource:0}: Error finding container 9ec423fb0b8cbc0be567cbff0bae37f5227a788dfc3af0fc6b7e9cd93134462d: Status 404 returned error can't find the container with id 9ec423fb0b8cbc0be567cbff0bae37f5227a788dfc3af0fc6b7e9cd93134462d Nov 03 09:12:43.303520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:43.300720 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7"] Nov 03 09:12:43.759902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:43.759861 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" event={"ID":"0ad07039-b190-421f-8c5f-32be14742a24","Type":"ContainerStarted","Data":"5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a"} Nov 03 09:12:43.760404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:43.759929 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" event={"ID":"0ad07039-b190-421f-8c5f-32be14742a24","Type":"ContainerStarted","Data":"ae961c1d1213617434409ed8ed704500fa14d9fa79d0ed99f902c6f575a1b6bc"} Nov 03 09:12:43.760404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:43.759975 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:12:43.761906 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:43.761870 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" event={"ID":"28e9f4e5-1c70-40c9-a56d-dff029b79b11","Type":"ContainerStarted","Data":"7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76"} Nov 03 09:12:43.762019 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:43.761926 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" event={"ID":"28e9f4e5-1c70-40c9-a56d-dff029b79b11","Type":"ContainerStarted","Data":"9ec423fb0b8cbc0be567cbff0bae37f5227a788dfc3af0fc6b7e9cd93134462d"} Nov 03 09:12:43.762019 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:43.761958 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:12:43.877318 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:43.877257 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" podStartSLOduration=1.877237355 podStartE2EDuration="1.877237355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:12:43.876577905 +0000 UTC m=+1996.801520123" watchObservedRunningTime="2026-11-03 09:12:43.877237355 +0000 UTC m=+1996.802179571" Nov 03 09:12:44.002527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:44.001137 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" podStartSLOduration=2.001112259 podStartE2EDuration="2.001112259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:12:43.994817047 +0000 UTC m=+1996.919759265" watchObservedRunningTime="2026-11-03 09:12:44.001112259 +0000 UTC m=+1996.926054477" Nov 03 09:12:46.904108 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:12:46.904065 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:12:54.774423 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:54.774388 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:12:54.774931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:12:54.774670 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:13:07.641609 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:07.641530 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9"] Nov 03 09:13:07.753654 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:07.753617 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt"] Nov 03 09:13:12.641965 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:12.641913 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" podUID="215363c2-4ede-4a60-b066-d9d500f2c9c1" containerName="payload-processing" containerID="cri-o://a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114" gracePeriod=25 Nov 03 09:13:12.754344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:12.754292 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" podUID="6ce18997-659f-4612-8c93-d660f3a21673" containerName="payload-pre-processing" containerID="cri-o://e55d1c787693fa4b2e6566ca1835ad832420fb8bfa6ce4ccaeacba8c0a09d29b" gracePeriod=25 Nov 03 09:13:12.969781 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:12.969749 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:13:13.030958 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.030819 5015 generic.go:356] "Generic (PLEG): container finished" podID="215363c2-4ede-4a60-b066-d9d500f2c9c1" containerID="a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114" exitCode=0 Nov 03 09:13:13.030958 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.030921 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" Nov 03 09:13:13.031188 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.030927 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" event={"ID":"215363c2-4ede-4a60-b066-d9d500f2c9c1","Type":"ContainerDied","Data":"a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114"} Nov 03 09:13:13.031188 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.031177 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9" event={"ID":"215363c2-4ede-4a60-b066-d9d500f2c9c1","Type":"ContainerDied","Data":"7f7e8bca4a0c51004e969d6b496628e5602c31b0cdf79ead367524e23d6a9582"} Nov 03 09:13:13.031276 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.031209 5015 scope.go:122] "RemoveContainer" containerID="a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114" Nov 03 09:13:13.032920 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.032899 5015 generic.go:356] "Generic (PLEG): container finished" podID="6ce18997-659f-4612-8c93-d660f3a21673" containerID="e55d1c787693fa4b2e6566ca1835ad832420fb8bfa6ce4ccaeacba8c0a09d29b" exitCode=0 Nov 03 09:13:13.033052 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.033022 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" event={"ID":"6ce18997-659f-4612-8c93-d660f3a21673","Type":"ContainerDied","Data":"e55d1c787693fa4b2e6566ca1835ad832420fb8bfa6ce4ccaeacba8c0a09d29b"} Nov 03 09:13:13.057910 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.057883 5015 scope.go:122] "RemoveContainer" containerID="a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114" Nov 03 09:13:13.058321 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:13:13.058299 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114\": container with ID starting with a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114 not found: ID does not exist" containerID="a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114" Nov 03 09:13:13.058424 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.058334 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114"} err="failed to get container status \"a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114\": rpc error: code = NotFound desc = could not find container \"a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114\": container with ID starting with a9f4d24be1c04b66ba97fe6209f4f8ef787f6e033abcf928cbd979b92d868114 not found: ID does not exist" Nov 03 09:13:13.082813 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.082785 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:13:13.104065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.104034 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/215363c2-4ede-4a60-b066-d9d500f2c9c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215363c2-4ede-4a60-b066-d9d500f2c9c1-plugins-config-volume\") pod \"215363c2-4ede-4a60-b066-d9d500f2c9c1\" (UID: \"215363c2-4ede-4a60-b066-d9d500f2c9c1\") " Nov 03 09:13:13.104228 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.104077 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/215363c2-4ede-4a60-b066-d9d500f2c9c1-kube-api-access-jx2x7\" (UniqueName: \"kubernetes.io/projected/215363c2-4ede-4a60-b066-d9d500f2c9c1-kube-api-access-jx2x7\") pod \"215363c2-4ede-4a60-b066-d9d500f2c9c1\" (UID: \"215363c2-4ede-4a60-b066-d9d500f2c9c1\") " Nov 03 09:13:13.104526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.104468 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215363c2-4ede-4a60-b066-d9d500f2c9c1-plugins-config-volume" pod "215363c2-4ede-4a60-b066-d9d500f2c9c1" (UID: "215363c2-4ede-4a60-b066-d9d500f2c9c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:13:13.104809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.104786 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215363c2-4ede-4a60-b066-d9d500f2c9c1-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:13.106583 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.106558 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215363c2-4ede-4a60-b066-d9d500f2c9c1-kube-api-access-jx2x7" pod "215363c2-4ede-4a60-b066-d9d500f2c9c1" (UID: "215363c2-4ede-4a60-b066-d9d500f2c9c1"). InnerVolumeSpecName "kube-api-access-jx2x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:13:13.205545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.205442 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ce18997-659f-4612-8c93-d660f3a21673-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce18997-659f-4612-8c93-d660f3a21673-plugins-config-volume\") pod \"6ce18997-659f-4612-8c93-d660f3a21673\" (UID: \"6ce18997-659f-4612-8c93-d660f3a21673\") " Nov 03 09:13:13.205545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.205531 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ce18997-659f-4612-8c93-d660f3a21673-kube-api-access-bs6w5\" (UniqueName: \"kubernetes.io/projected/6ce18997-659f-4612-8c93-d660f3a21673-kube-api-access-bs6w5\") pod \"6ce18997-659f-4612-8c93-d660f3a21673\" (UID: \"6ce18997-659f-4612-8c93-d660f3a21673\") " Nov 03 09:13:13.205922 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.205893 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce18997-659f-4612-8c93-d660f3a21673-plugins-config-volume" pod "6ce18997-659f-4612-8c93-d660f3a21673" (UID: "6ce18997-659f-4612-8c93-d660f3a21673"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:13:13.206027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.205901 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx2x7\" (UniqueName: \"kubernetes.io/projected/215363c2-4ede-4a60-b066-d9d500f2c9c1-kube-api-access-jx2x7\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:13.207923 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.207900 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce18997-659f-4612-8c93-d660f3a21673-kube-api-access-bs6w5" pod "6ce18997-659f-4612-8c93-d660f3a21673" (UID: "6ce18997-659f-4612-8c93-d660f3a21673"). InnerVolumeSpecName "kube-api-access-bs6w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:13:13.307125 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.307089 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce18997-659f-4612-8c93-d660f3a21673-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:13.307125 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.307125 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs6w5\" (UniqueName: \"kubernetes.io/projected/6ce18997-659f-4612-8c93-d660f3a21673-kube-api-access-bs6w5\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:13.453283 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.453244 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9"] Nov 03 09:13:13.456873 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:13.456800 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-6484b4d6bc-pdzz9"] Nov 03 09:13:14.044028 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:14.043995 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" Nov 03 09:13:14.044455 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:14.044003 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt" event={"ID":"6ce18997-659f-4612-8c93-d660f3a21673","Type":"ContainerDied","Data":"676663c4ca949fdab943857f412725ea77490de319f4e4438e16f04db0c3855c"} Nov 03 09:13:14.044531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:14.044497 5015 scope.go:122] "RemoveContainer" containerID="e55d1c787693fa4b2e6566ca1835ad832420fb8bfa6ce4ccaeacba8c0a09d29b" Nov 03 09:13:14.215917 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:14.215885 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt"] Nov 03 09:13:14.222173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:14.222138 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-6778ffc8ff-jsdbt"] Nov 03 09:13:15.205148 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:15.205111 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="215363c2-4ede-4a60-b066-d9d500f2c9c1" path="/var/lib/kubelet/pods/215363c2-4ede-4a60-b066-d9d500f2c9c1/volumes" Nov 03 09:13:15.205573 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:15.205561 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ce18997-659f-4612-8c93-d660f3a21673" path="/var/lib/kubelet/pods/6ce18997-659f-4612-8c93-d660f3a21673/volumes" Nov 03 09:13:16.902775 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:13:16.902743 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:13:18.755549 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.755515 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj"] Nov 03 09:13:18.756208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.756193 5015 cpu_manager.go:414] "containerMap: removing container" podUID="215363c2-4ede-4a60-b066-d9d500f2c9c1" containerName="payload-processing" Nov 03 09:13:18.756272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.756210 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="215363c2-4ede-4a60-b066-d9d500f2c9c1" containerName="payload-processing" Nov 03 09:13:18.756272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.756229 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6ce18997-659f-4612-8c93-d660f3a21673" containerName="payload-pre-processing" Nov 03 09:13:18.756272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.756235 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ce18997-659f-4612-8c93-d660f3a21673" containerName="payload-pre-processing" Nov 03 09:13:18.756373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.756333 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="215363c2-4ede-4a60-b066-d9d500f2c9c1" containerName="payload-processing" Nov 03 09:13:18.756373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.756347 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ce18997-659f-4612-8c93-d660f3a21673" containerName="payload-pre-processing" Nov 03 09:13:18.760069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.760052 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:18.819884 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.819852 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj"] Nov 03 09:13:18.865447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.865419 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-133b31-serving-cert\"" Nov 03 09:13:18.876183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.876161 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7blx\" (UniqueName: \"kubernetes.io/projected/f02a39f0-92d4-4963-8be6-18dbe4da520e-kube-api-access-n7blx\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:18.876290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.876190 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:18.876338 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.876317 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-metrics-certs\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:18.876387 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.876374 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02a39f0-92d4-4963-8be6-18dbe4da520e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:18.977395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.977361 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7blx\" (UniqueName: \"kubernetes.io/projected/f02a39f0-92d4-4963-8be6-18dbe4da520e-kube-api-access-n7blx\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:18.977605 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.977402 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:18.977605 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.977535 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-metrics-certs\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:18.977836 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:13:18.977598 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-serving-cert: secret "maas-api-e2e-praxis-a-133b31-serving-cert" not found Nov 03 09:13:18.977836 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.977632 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02a39f0-92d4-4963-8be6-18dbe4da520e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:18.977836 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:13:18.977675 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-maas-api-tls podName:f02a39f0-92d4-4963-8be6-18dbe4da520e nodeName:}" failed. No retries permitted until 2026-11-03 09:13:19.477652604 +0000 UTC m=+2032.402594825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-maas-api-tls") pod "maas-api-e2e-praxis-a-133b31-86f755d454-724sj" (UID: "f02a39f0-92d4-4963-8be6-18dbe4da520e") : secret "maas-api-e2e-praxis-a-133b31-serving-cert" not found Nov 03 09:13:18.978443 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.978419 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02a39f0-92d4-4963-8be6-18dbe4da520e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:18.981282 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:18.981257 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-metrics-certs\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:19.037248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:19.037175 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7blx\" (UniqueName: \"kubernetes.io/projected/f02a39f0-92d4-4963-8be6-18dbe4da520e-kube-api-access-n7blx\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:19.484983 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:19.484945 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:19.487699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:19.487672 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-133b31-86f755d454-724sj\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:19.674498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:19.674446 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:20.063792 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:13:20.063748 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02a39f0_92d4_4963_8be6_18dbe4da520e.slice/crio-bb3c1d9f7726b95295a4cd5546b89a23658a401f9a7ea9679a2a08b4d1fd530f WatchSource:0}: Error finding container bb3c1d9f7726b95295a4cd5546b89a23658a401f9a7ea9679a2a08b4d1fd530f: Status 404 returned error can't find the container with id bb3c1d9f7726b95295a4cd5546b89a23658a401f9a7ea9679a2a08b4d1fd530f Nov 03 09:13:20.066899 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:20.066874 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj"] Nov 03 09:13:20.097119 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:20.097078 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" event={"ID":"f02a39f0-92d4-4963-8be6-18dbe4da520e","Type":"ContainerStarted","Data":"bb3c1d9f7726b95295a4cd5546b89a23658a401f9a7ea9679a2a08b4d1fd530f"} Nov 03 09:13:21.106218 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:21.106180 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" event={"ID":"f02a39f0-92d4-4963-8be6-18dbe4da520e","Type":"ContainerStarted","Data":"71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88"} Nov 03 09:13:21.106710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:21.106270 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:21.236801 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:21.236750 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" podStartSLOduration=2.385795669 podStartE2EDuration="3.236737053s" totalImagesPullingTime="850.941384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:13:21.233703913 +0000 UTC m=+2034.158646132" watchObservedRunningTime="2026-11-03 09:13:21.236737053 +0000 UTC m=+2034.161679270" Nov 03 09:13:27.119533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:27.119499 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:13:39.501709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:39.501672 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5"] Nov 03 09:13:39.507525 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:39.507505 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:13:39.576894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:39.576860 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5"] Nov 03 09:13:39.609381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:39.609337 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-77da2a\"" Nov 03 09:13:39.609381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:39.609360 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-77da2a-dockercfg-jl8ht\"" Nov 03 09:13:39.674302 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:39.674258 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zppz\" (UniqueName: \"kubernetes.io/projected/615d46d0-2300-496e-aa08-9982e5dc99b2-kube-api-access-2zppz\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5\" (UID: \"615d46d0-2300-496e-aa08-9982e5dc99b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:13:39.674552 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:39.674525 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615d46d0-2300-496e-aa08-9982e5dc99b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5\" (UID: \"615d46d0-2300-496e-aa08-9982e5dc99b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:13:39.776003 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:39.775918 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zppz\" (UniqueName: \"kubernetes.io/projected/615d46d0-2300-496e-aa08-9982e5dc99b2-kube-api-access-2zppz\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5\" (UID: \"615d46d0-2300-496e-aa08-9982e5dc99b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:13:39.776167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:39.776118 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615d46d0-2300-496e-aa08-9982e5dc99b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5\" (UID: \"615d46d0-2300-496e-aa08-9982e5dc99b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:13:39.776921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:39.776900 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615d46d0-2300-496e-aa08-9982e5dc99b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5\" (UID: \"615d46d0-2300-496e-aa08-9982e5dc99b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:13:39.835458 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:39.835425 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zppz\" (UniqueName: \"kubernetes.io/projected/615d46d0-2300-496e-aa08-9982e5dc99b2-kube-api-access-2zppz\") pod \"payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5\" (UID: \"615d46d0-2300-496e-aa08-9982e5dc99b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:13:40.123142 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.123062 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:13:40.291874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.291841 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9"] Nov 03 09:13:40.297440 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.297410 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:13:40.372234 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.372202 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9"] Nov 03 09:13:40.388847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.388752 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7468ac-f9fb-43ba-b333-6644840e9919-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9\" (UID: \"ea7468ac-f9fb-43ba-b333-6644840e9919\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:13:40.388847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.388839 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqf5d\" (UniqueName: \"kubernetes.io/projected/ea7468ac-f9fb-43ba-b333-6644840e9919-kube-api-access-xqf5d\") pod \"payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9\" (UID: \"ea7468ac-f9fb-43ba-b333-6644840e9919\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:13:40.490746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.490714 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7468ac-f9fb-43ba-b333-6644840e9919-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9\" (UID: \"ea7468ac-f9fb-43ba-b333-6644840e9919\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:13:40.490931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.490774 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqf5d\" (UniqueName: \"kubernetes.io/projected/ea7468ac-f9fb-43ba-b333-6644840e9919-kube-api-access-xqf5d\") pod \"payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9\" (UID: \"ea7468ac-f9fb-43ba-b333-6644840e9919\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:13:40.491401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.491379 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7468ac-f9fb-43ba-b333-6644840e9919-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9\" (UID: \"ea7468ac-f9fb-43ba-b333-6644840e9919\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:13:40.565169 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.565133 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5"] Nov 03 09:13:40.565620 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.565162 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqf5d\" (UniqueName: \"kubernetes.io/projected/ea7468ac-f9fb-43ba-b333-6644840e9919-kube-api-access-xqf5d\") pod \"payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9\" (UID: \"ea7468ac-f9fb-43ba-b333-6644840e9919\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:13:40.566687 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:13:40.566660 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615d46d0_2300_496e_aa08_9982e5dc99b2.slice/crio-0d55def18f7278b529611febd1dcc5b12c290cfd11047ec6ef2a1eb98bec85b9 WatchSource:0}: Error finding container 0d55def18f7278b529611febd1dcc5b12c290cfd11047ec6ef2a1eb98bec85b9: Status 404 returned error can't find the container with id 0d55def18f7278b529611febd1dcc5b12c290cfd11047ec6ef2a1eb98bec85b9 Nov 03 09:13:40.624902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:40.624877 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:13:41.066712 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:41.066561 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9"] Nov 03 09:13:41.068764 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:13:41.068733 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7468ac_f9fb_43ba_b333_6644840e9919.slice/crio-4280beb809341b3f423344cb17721ec7c25b9d3bca48360d0f997bdc22856746 WatchSource:0}: Error finding container 4280beb809341b3f423344cb17721ec7c25b9d3bca48360d0f997bdc22856746: Status 404 returned error can't find the container with id 4280beb809341b3f423344cb17721ec7c25b9d3bca48360d0f997bdc22856746 Nov 03 09:13:41.288628 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:41.288548 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" event={"ID":"615d46d0-2300-496e-aa08-9982e5dc99b2","Type":"ContainerStarted","Data":"3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca"} Nov 03 09:13:41.288946 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:41.288913 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" event={"ID":"615d46d0-2300-496e-aa08-9982e5dc99b2","Type":"ContainerStarted","Data":"0d55def18f7278b529611febd1dcc5b12c290cfd11047ec6ef2a1eb98bec85b9"} Nov 03 09:13:41.289075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:41.288961 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:13:41.290568 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:41.290544 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" event={"ID":"ea7468ac-f9fb-43ba-b333-6644840e9919","Type":"ContainerStarted","Data":"d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e"} Nov 03 09:13:41.290674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:41.290588 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" event={"ID":"ea7468ac-f9fb-43ba-b333-6644840e9919","Type":"ContainerStarted","Data":"4280beb809341b3f423344cb17721ec7c25b9d3bca48360d0f997bdc22856746"} Nov 03 09:13:41.290674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:41.290636 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:13:41.410196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:41.410085 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" podStartSLOduration=2.410067202 podStartE2EDuration="2.410067202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:13:41.406918657 +0000 UTC m=+2054.331860875" watchObservedRunningTime="2026-11-03 09:13:41.410067202 +0000 UTC m=+2054.335009420" Nov 03 09:13:41.552270 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:41.552197 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" podStartSLOduration=1.5521596930000001 podStartE2EDuration="1.552159693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:13:41.55078235 +0000 UTC m=+2054.475724568" watchObservedRunningTime="2026-11-03 09:13:41.552159693 +0000 UTC m=+2054.477101913" Nov 03 09:13:46.906246 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:13:46.906204 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:13:47.437939 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:47.437904 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68_56530cb6-7d91-469a-96cb-4049f1ec5e5a/payload-processing/1.log" Nov 03 09:13:47.667430 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:47.667399 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68_56530cb6-7d91-469a-96cb-4049f1ec5e5a/payload-processing/1.log" Nov 03 09:13:48.348840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:48.348806 5015 generic.go:356] "Generic (PLEG): container finished" podID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerID="17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555" exitCode=0 Nov 03 09:13:48.349421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:48.348876 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" event={"ID":"2510f27d-c29c-43b0-bf06-43dc2d174f56","Type":"ContainerDied","Data":"17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555"} Nov 03 09:13:48.349421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:48.349327 5015 scope.go:122] "RemoveContainer" containerID="17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555" Nov 03 09:13:48.351224 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:48.351186 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:13:48.351333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:48.351282 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:13:48.351333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:48.351326 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:13:49.370964 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:49.370926 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" event={"ID":"2510f27d-c29c-43b0-bf06-43dc2d174f56","Type":"ContainerStarted","Data":"b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75"} Nov 03 09:13:49.601647 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:49.601618 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:13:49.601935 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:49.601913 5015 patch_prober.go:29] interesting pod/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:13:49.601986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:49.601972 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 09:13:50.562502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:50.562348 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4"] Nov 03 09:13:50.563090 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:50.562688 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" podUID="8d46a006-7770-41da-9644-5c70213f8dd7" containerName="maas-api" containerID="cri-o://c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f" gracePeriod=30 Nov 03 09:13:50.601780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:50.601749 5015 patch_prober.go:29] interesting pod/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:13:50.601925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:50.601821 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 09:13:50.922900 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:50.922868 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:13:51.064411 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.064365 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs"] Nov 03 09:13:51.064682 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.064653 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" podUID="2b3f774c-9781-4bf6-8594-fd0210ad8256" containerName="payload-processing" containerID="cri-o://e68d83ec39ff5850d087287a7046699d85adfb47d13ce4dc799ba4d2042309ac" gracePeriod=30 Nov 03 09:13:51.098074 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.098044 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d46a006-7770-41da-9644-5c70213f8dd7-kube-api-access-b7hsp\" (UniqueName: \"kubernetes.io/projected/8d46a006-7770-41da-9644-5c70213f8dd7-kube-api-access-b7hsp\") pod \"8d46a006-7770-41da-9644-5c70213f8dd7\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " Nov 03 09:13:51.098217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.098200 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-metrics-certs\") pod \"8d46a006-7770-41da-9644-5c70213f8dd7\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " Nov 03 09:13:51.098323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.098306 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d46a006-7770-41da-9644-5c70213f8dd7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d46a006-7770-41da-9644-5c70213f8dd7-openshift-service-ca\") pod \"8d46a006-7770-41da-9644-5c70213f8dd7\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " Nov 03 09:13:51.098441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.098418 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-maas-api-tls\") pod \"8d46a006-7770-41da-9644-5c70213f8dd7\" (UID: \"8d46a006-7770-41da-9644-5c70213f8dd7\") " Nov 03 09:13:51.098697 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.098673 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d46a006-7770-41da-9644-5c70213f8dd7-openshift-service-ca" pod "8d46a006-7770-41da-9644-5c70213f8dd7" (UID: "8d46a006-7770-41da-9644-5c70213f8dd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:13:51.099278 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.099256 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d46a006-7770-41da-9644-5c70213f8dd7-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:51.100378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.100355 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d46a006-7770-41da-9644-5c70213f8dd7-kube-api-access-b7hsp" pod "8d46a006-7770-41da-9644-5c70213f8dd7" (UID: "8d46a006-7770-41da-9644-5c70213f8dd7"). InnerVolumeSpecName "kube-api-access-b7hsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:13:51.100518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.100493 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-metrics-certs" pod "8d46a006-7770-41da-9644-5c70213f8dd7" (UID: "8d46a006-7770-41da-9644-5c70213f8dd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:13:51.100774 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.100755 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-maas-api-tls" pod "8d46a006-7770-41da-9644-5c70213f8dd7" (UID: "8d46a006-7770-41da-9644-5c70213f8dd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:13:51.205520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.202797 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:51.205520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.202835 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7hsp\" (UniqueName: \"kubernetes.io/projected/8d46a006-7770-41da-9644-5c70213f8dd7-kube-api-access-b7hsp\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:51.205520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.202852 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d46a006-7770-41da-9644-5c70213f8dd7-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:51.391219 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.391178 5015 generic.go:356] "Generic (PLEG): container finished" podID="2b3f774c-9781-4bf6-8594-fd0210ad8256" containerID="e68d83ec39ff5850d087287a7046699d85adfb47d13ce4dc799ba4d2042309ac" exitCode=0 Nov 03 09:13:51.391219 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.391208 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" event={"ID":"2b3f774c-9781-4bf6-8594-fd0210ad8256","Type":"ContainerDied","Data":"e68d83ec39ff5850d087287a7046699d85adfb47d13ce4dc799ba4d2042309ac"} Nov 03 09:13:51.393156 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.393131 5015 generic.go:356] "Generic (PLEG): container finished" podID="8d46a006-7770-41da-9644-5c70213f8dd7" containerID="c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f" exitCode=0 Nov 03 09:13:51.393303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.393183 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" Nov 03 09:13:51.393303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.393290 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" event={"ID":"8d46a006-7770-41da-9644-5c70213f8dd7","Type":"ContainerDied","Data":"c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f"} Nov 03 09:13:51.393404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.393332 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4" event={"ID":"8d46a006-7770-41da-9644-5c70213f8dd7","Type":"ContainerDied","Data":"816f97611834177d73b5a904c06389ff02a22cd4212024034e89b4d00a5ae4d5"} Nov 03 09:13:51.393404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.393362 5015 scope.go:122] "RemoveContainer" containerID="c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f" Nov 03 09:13:51.405522 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.405458 5015 scope.go:122] "RemoveContainer" containerID="c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f" Nov 03 09:13:51.405869 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:13:51.405841 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f\": container with ID starting with c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f not found: ID does not exist" containerID="c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f" Nov 03 09:13:51.405945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.405883 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f"} err="failed to get container status \"c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f\": rpc error: code = NotFound desc = could not find container \"c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f\": container with ID starting with c783c430f8f2ccc0c4524b3bf729e4c54e63f248524dd1401103ff63e4c56a5f not found: ID does not exist" Nov 03 09:13:51.420164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.420145 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:13:51.515027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.514993 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b3f774c-9781-4bf6-8594-fd0210ad8256-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3f774c-9781-4bf6-8594-fd0210ad8256-plugins-config-volume\") pod \"2b3f774c-9781-4bf6-8594-fd0210ad8256\" (UID: \"2b3f774c-9781-4bf6-8594-fd0210ad8256\") " Nov 03 09:13:51.515205 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.515163 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b3f774c-9781-4bf6-8594-fd0210ad8256-kube-api-access-9bjhf\" (UniqueName: \"kubernetes.io/projected/2b3f774c-9781-4bf6-8594-fd0210ad8256-kube-api-access-9bjhf\") pod \"2b3f774c-9781-4bf6-8594-fd0210ad8256\" (UID: \"2b3f774c-9781-4bf6-8594-fd0210ad8256\") " Nov 03 09:13:51.515436 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.515409 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3f774c-9781-4bf6-8594-fd0210ad8256-plugins-config-volume" pod "2b3f774c-9781-4bf6-8594-fd0210ad8256" (UID: "2b3f774c-9781-4bf6-8594-fd0210ad8256"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:13:51.515781 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.515754 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3f774c-9781-4bf6-8594-fd0210ad8256-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:51.517582 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.517559 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3f774c-9781-4bf6-8594-fd0210ad8256-kube-api-access-9bjhf" pod "2b3f774c-9781-4bf6-8594-fd0210ad8256" (UID: "2b3f774c-9781-4bf6-8594-fd0210ad8256"). InnerVolumeSpecName "kube-api-access-9bjhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:13:51.605839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.601792 5015 patch_prober.go:29] interesting pod/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:13:51.605839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.601870 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 09:13:51.654503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.646559 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bjhf\" (UniqueName: \"kubernetes.io/projected/2b3f774c-9781-4bf6-8594-fd0210ad8256-kube-api-access-9bjhf\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:51.721593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.721562 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4"] Nov 03 09:13:51.728505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:51.726660 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ae3db-7cbb5c4f6b-8qdj4"] Nov 03 09:13:52.300421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:52.300391 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:13:52.300634 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:52.300465 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:13:52.409548 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:52.409506 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" Nov 03 09:13:52.409548 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:52.409517 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs" event={"ID":"2b3f774c-9781-4bf6-8594-fd0210ad8256","Type":"ContainerDied","Data":"e4a225de316eca7ff75bdb2e0dd5976f279e68a7bce4a2724b89cba153c63896"} Nov 03 09:13:52.409802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:52.409606 5015 scope.go:122] "RemoveContainer" containerID="e68d83ec39ff5850d087287a7046699d85adfb47d13ce4dc799ba4d2042309ac" Nov 03 09:13:52.601635 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:52.601555 5015 patch_prober.go:29] interesting pod/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:13:52.601635 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:52.601618 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 09:13:52.685508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:52.685432 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs"] Nov 03 09:13:52.690315 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:52.690286 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ae3db-68bcff887b-qwvxs"] Nov 03 09:13:53.205690 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:53.205655 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b3f774c-9781-4bf6-8594-fd0210ad8256" path="/var/lib/kubelet/pods/2b3f774c-9781-4bf6-8594-fd0210ad8256/volumes" Nov 03 09:13:53.206199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:53.206183 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d46a006-7770-41da-9644-5c70213f8dd7" path="/var/lib/kubelet/pods/8d46a006-7770-41da-9644-5c70213f8dd7/volumes" Nov 03 09:13:53.601845 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:53.601805 5015 patch_prober.go:29] interesting pod/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:13:53.602031 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:53.601862 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 09:13:54.683767 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:54.683726 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:13:54.684267 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:54.684162 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:13:54.685409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:54.685389 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:13:59.016298 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:59.016264 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr"] Nov 03 09:13:59.016822 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:13:59.016513 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" containerID="cri-o://e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d" gracePeriod=30 Nov 03 09:14:03.030201 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:03.030163 5015 patch_prober.go:29] interesting pod/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:14:03.030660 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:03.030227 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:14:04.303155 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.303095 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:14:04.433861 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.433825 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-envoy\") pod \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " Nov 03 09:14:04.434040 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.433950 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-credential-socket\") pod \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " Nov 03 09:14:04.434040 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.433981 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-token\" (UniqueName: \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-token\") pod \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " Nov 03 09:14:04.434040 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.434025 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-kube-api-access-g462s\" (UniqueName: \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-kube-api-access-g462s\") pod \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " Nov 03 09:14:04.434206 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.434081 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-podinfo\") pod \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " Nov 03 09:14:04.434206 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.434102 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-data\") pod \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " Nov 03 09:14:04.434206 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.434150 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-certs\") pod \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " Nov 03 09:14:04.434372 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.434208 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istiod-ca-cert\") pod \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " Nov 03 09:14:04.434372 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.434248 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-socket\") pod \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\" (UID: \"11d1b378-6be0-4b3d-95ef-71b82d571d7f\") " Nov 03 09:14:04.434372 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.434269 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-credential-socket" pod "11d1b378-6be0-4b3d-95ef-71b82d571d7f" (UID: "11d1b378-6be0-4b3d-95ef-71b82d571d7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:04.434583 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.434523 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-data" pod "11d1b378-6be0-4b3d-95ef-71b82d571d7f" (UID: "11d1b378-6be0-4b3d-95ef-71b82d571d7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:04.434812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.434668 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-certs" pod "11d1b378-6be0-4b3d-95ef-71b82d571d7f" (UID: "11d1b378-6be0-4b3d-95ef-71b82d571d7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:04.434812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.434727 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-socket" pod "11d1b378-6be0-4b3d-95ef-71b82d571d7f" (UID: "11d1b378-6be0-4b3d-95ef-71b82d571d7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:04.434812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.434797 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istiod-ca-cert" pod "11d1b378-6be0-4b3d-95ef-71b82d571d7f" (UID: "11d1b378-6be0-4b3d-95ef-71b82d571d7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:14:04.435316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.435293 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:04.435381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.435322 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:04.435381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.435338 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:04.435381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.435355 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:04.435381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.435369 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:04.436355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.436324 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-envoy" pod "11d1b378-6be0-4b3d-95ef-71b82d571d7f" (UID: "11d1b378-6be0-4b3d-95ef-71b82d571d7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:04.436684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.436658 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-token" pod "11d1b378-6be0-4b3d-95ef-71b82d571d7f" (UID: "11d1b378-6be0-4b3d-95ef-71b82d571d7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:14:04.436782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.436686 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-kube-api-access-g462s" pod "11d1b378-6be0-4b3d-95ef-71b82d571d7f" (UID: "11d1b378-6be0-4b3d-95ef-71b82d571d7f"). InnerVolumeSpecName "kube-api-access-g462s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:14:04.437096 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.437072 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-podinfo" pod "11d1b378-6be0-4b3d-95ef-71b82d571d7f" (UID: "11d1b378-6be0-4b3d-95ef-71b82d571d7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:14:04.506568 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.506528 5015 generic.go:356] "Generic (PLEG): container finished" podID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerID="e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d" exitCode=0 Nov 03 09:14:04.506740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.506600 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" Nov 03 09:14:04.506740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.506611 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" event={"ID":"11d1b378-6be0-4b3d-95ef-71b82d571d7f","Type":"ContainerDied","Data":"e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d"} Nov 03 09:14:04.506740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.506664 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr" event={"ID":"11d1b378-6be0-4b3d-95ef-71b82d571d7f","Type":"ContainerDied","Data":"f9db319a77154d9a9071ea8adb784662abc2606a9b4307ea8db055eacb9b14e9"} Nov 03 09:14:04.506740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.506692 5015 scope.go:122] "RemoveContainer" containerID="e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d" Nov 03 09:14:04.518996 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.518971 5015 scope.go:122] "RemoveContainer" containerID="389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49" Nov 03 09:14:04.529676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.529656 5015 scope.go:122] "RemoveContainer" containerID="e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d" Nov 03 09:14:04.529925 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:14:04.529907 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d\": container with ID starting with e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d not found: ID does not exist" containerID="e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d" Nov 03 09:14:04.529981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.529932 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d"} err="failed to get container status \"e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d\": rpc error: code = NotFound desc = could not find container \"e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d\": container with ID starting with e2b542fb813e1e7b7d9ae5232d22d3f8dbe6d2df8b63ed34c6dcc910cfced74d not found: ID does not exist" Nov 03 09:14:04.529981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.529955 5015 scope.go:122] "RemoveContainer" containerID="389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49" Nov 03 09:14:04.530182 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:14:04.530162 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49\": container with ID starting with 389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49 not found: ID does not exist" containerID="389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49" Nov 03 09:14:04.530238 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.530194 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49"} err="failed to get container status \"389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49\": rpc error: code = NotFound desc = could not find container \"389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49\": container with ID starting with 389cf0da32a7c58cd7d8b4fef54c85a144a47102b464507526db7e2c8e23ef49 not found: ID does not exist" Nov 03 09:14:04.537029 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.537006 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:04.537163 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.537146 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:04.537237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.537168 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g462s\" (UniqueName: \"kubernetes.io/projected/11d1b378-6be0-4b3d-95ef-71b82d571d7f-kube-api-access-g462s\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:04.537237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.537184 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d1b378-6be0-4b3d-95ef-71b82d571d7f-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:04.680015 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.679924 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr"] Nov 03 09:14:04.690827 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:04.690798 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ae3db-openshift-default-f58fdfffc-bt4zr"] Nov 03 09:14:05.205944 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:05.205910 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" path="/var/lib/kubelet/pods/11d1b378-6be0-4b3d-95ef-71b82d571d7f/volumes" Nov 03 09:14:05.683544 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:05.680691 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn"] Nov 03 09:14:10.484047 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.483991 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb"] Nov 03 09:14:10.485570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485533 5015 cpu_manager.go:414] "containerMap: removing container" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" Nov 03 09:14:10.485570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485564 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" Nov 03 09:14:10.485784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485583 5015 cpu_manager.go:414] "containerMap: removing container" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" Nov 03 09:14:10.485784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485591 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" Nov 03 09:14:10.485784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485611 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8d46a006-7770-41da-9644-5c70213f8dd7" containerName="maas-api" Nov 03 09:14:10.485784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485617 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d46a006-7770-41da-9644-5c70213f8dd7" containerName="maas-api" Nov 03 09:14:10.485784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485632 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2b3f774c-9781-4bf6-8594-fd0210ad8256" containerName="payload-processing" Nov 03 09:14:10.485784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485636 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b3f774c-9781-4bf6-8594-fd0210ad8256" containerName="payload-processing" Nov 03 09:14:10.485784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485770 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" Nov 03 09:14:10.485784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485778 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d1b378-6be0-4b3d-95ef-71b82d571d7f" containerName="istio-proxy" Nov 03 09:14:10.485784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485786 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d46a006-7770-41da-9644-5c70213f8dd7" containerName="maas-api" Nov 03 09:14:10.486217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.485795 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b3f774c-9781-4bf6-8594-fd0210ad8256" containerName="payload-processing" Nov 03 09:14:10.491152 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.491132 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:14:10.531892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.531865 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb477\" (UniqueName: \"kubernetes.io/projected/569b803e-986a-47b3-8552-3f80dd115e23-kube-api-access-gb477\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb\" (UID: \"569b803e-986a-47b3-8552-3f80dd115e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:14:10.532076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.532056 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569b803e-986a-47b3-8552-3f80dd115e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb\" (UID: \"569b803e-986a-47b3-8552-3f80dd115e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:14:10.571852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.571816 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb"] Nov 03 09:14:10.592291 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.592244 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c78f2b\"" Nov 03 09:14:10.592465 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.592333 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c78f2b-dockercfg-7kwbb\"" Nov 03 09:14:10.633030 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.632992 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569b803e-986a-47b3-8552-3f80dd115e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb\" (UID: \"569b803e-986a-47b3-8552-3f80dd115e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:14:10.633197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.633086 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb477\" (UniqueName: \"kubernetes.io/projected/569b803e-986a-47b3-8552-3f80dd115e23-kube-api-access-gb477\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb\" (UID: \"569b803e-986a-47b3-8552-3f80dd115e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:14:10.633717 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.633698 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569b803e-986a-47b3-8552-3f80dd115e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb\" (UID: \"569b803e-986a-47b3-8552-3f80dd115e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:14:10.692390 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.692356 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb477\" (UniqueName: \"kubernetes.io/projected/569b803e-986a-47b3-8552-3f80dd115e23-kube-api-access-gb477\") pod \"payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb\" (UID: \"569b803e-986a-47b3-8552-3f80dd115e23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:14:10.810838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:10.810743 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:14:11.228573 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.228513 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb"] Nov 03 09:14:11.229203 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:14:11.229173 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569b803e_986a_47b3_8552_3f80dd115e23.slice/crio-b24029ca7d69da77dd774c84f79ca4b029fbdd57cec3dbb13e82ebce05373572 WatchSource:0}: Error finding container b24029ca7d69da77dd774c84f79ca4b029fbdd57cec3dbb13e82ebce05373572: Status 404 returned error can't find the container with id b24029ca7d69da77dd774c84f79ca4b029fbdd57cec3dbb13e82ebce05373572 Nov 03 09:14:11.266736 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.266703 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl"] Nov 03 09:14:11.272637 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.272603 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:14:11.371097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.371046 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl"] Nov 03 09:14:11.469586 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.468213 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxqq9\" (UniqueName: \"kubernetes.io/projected/24a0503c-ea10-46b4-91cc-665171c35c2c-kube-api-access-kxqq9\") pod \"payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl\" (UID: \"24a0503c-ea10-46b4-91cc-665171c35c2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:14:11.469586 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.468370 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a0503c-ea10-46b4-91cc-665171c35c2c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl\" (UID: \"24a0503c-ea10-46b4-91cc-665171c35c2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:14:11.569574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.569459 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxqq9\" (UniqueName: \"kubernetes.io/projected/24a0503c-ea10-46b4-91cc-665171c35c2c-kube-api-access-kxqq9\") pod \"payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl\" (UID: \"24a0503c-ea10-46b4-91cc-665171c35c2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:14:11.569574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.569554 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a0503c-ea10-46b4-91cc-665171c35c2c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl\" (UID: \"24a0503c-ea10-46b4-91cc-665171c35c2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:14:11.570408 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.570382 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a0503c-ea10-46b4-91cc-665171c35c2c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl\" (UID: \"24a0503c-ea10-46b4-91cc-665171c35c2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:14:11.575303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.575275 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" event={"ID":"569b803e-986a-47b3-8552-3f80dd115e23","Type":"ContainerStarted","Data":"060e445235ad1f1486f44ebca35574b048566cd92ba64c63e3b773514f15e412"} Nov 03 09:14:11.575446 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.575330 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" event={"ID":"569b803e-986a-47b3-8552-3f80dd115e23","Type":"ContainerStarted","Data":"b24029ca7d69da77dd774c84f79ca4b029fbdd57cec3dbb13e82ebce05373572"} Nov 03 09:14:11.575531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.575466 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:14:11.630510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.630445 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxqq9\" (UniqueName: \"kubernetes.io/projected/24a0503c-ea10-46b4-91cc-665171c35c2c-kube-api-access-kxqq9\") pod \"payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl\" (UID: \"24a0503c-ea10-46b4-91cc-665171c35c2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:14:11.695646 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.695579 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" podStartSLOduration=1.6955620850000002 podStartE2EDuration="1.695562085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:14:11.691675917 +0000 UTC m=+2084.616618138" watchObservedRunningTime="2026-11-03 09:14:11.695562085 +0000 UTC m=+2084.620504411" Nov 03 09:14:11.918933 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:11.918840 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:14:12.393822 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:12.393793 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl"] Nov 03 09:14:12.594496 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:12.593741 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" event={"ID":"24a0503c-ea10-46b4-91cc-665171c35c2c","Type":"ContainerStarted","Data":"e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e"} Nov 03 09:14:12.594496 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:12.593792 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" event={"ID":"24a0503c-ea10-46b4-91cc-665171c35c2c","Type":"ContainerStarted","Data":"f87a9ddde3c2c3adce79f50e8542f0181a1826d3843cd633e66794910e68d311"} Nov 03 09:14:12.594496 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:12.593841 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:14:12.715805 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:12.715706 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" podStartSLOduration=1.715692869 podStartE2EDuration="1.715692869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:14:12.713640859 +0000 UTC m=+2085.638583076" watchObservedRunningTime="2026-11-03 09:14:12.715692869 +0000 UTC m=+2085.640635085" Nov 03 09:14:16.904420 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:14:16.904372 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:14:20.666708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:20.666669 5015 generic.go:356] "Generic (PLEG): container finished" podID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerID="a84a08297e12eea2f078d0c1e2a2f8b206b6709c2d03c8db057130c1d05e00a4" exitCode=0 Nov 03 09:14:20.667267 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:20.666747 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" event={"ID":"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b","Type":"ContainerDied","Data":"a84a08297e12eea2f078d0c1e2a2f8b206b6709c2d03c8db057130c1d05e00a4"} Nov 03 09:14:20.667267 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:20.667214 5015 scope.go:122] "RemoveContainer" containerID="a84a08297e12eea2f078d0c1e2a2f8b206b6709c2d03c8db057130c1d05e00a4" Nov 03 09:14:20.669049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:20.669004 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:14:20.669162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:20.669103 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:14:20.669162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:20.669149 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:14:20.715791 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:20.715753 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" podUID="b72ebb8a-18db-4212-8513-59185daa0074" containerName="main" containerID="cri-o://e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc" gracePeriod=45 Nov 03 09:14:21.085801 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.085767 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:14:21.206704 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.206609 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-home\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-home\") pod \"b72ebb8a-18db-4212-8513-59185daa0074\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " Nov 03 09:14:21.206930 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.206708 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-model-cache\") pod \"b72ebb8a-18db-4212-8513-59185daa0074\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " Nov 03 09:14:21.207209 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.207179 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-model-cache" pod "b72ebb8a-18db-4212-8513-59185daa0074" (UID: "b72ebb8a-18db-4212-8513-59185daa0074"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:21.207376 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.207361 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-home" pod "b72ebb8a-18db-4212-8513-59185daa0074" (UID: "b72ebb8a-18db-4212-8513-59185daa0074"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:21.207951 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.207549 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-tmp-dir\") pod \"b72ebb8a-18db-4212-8513-59185daa0074\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " Nov 03 09:14:21.207951 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.207765 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-tmp-dir" pod "b72ebb8a-18db-4212-8513-59185daa0074" (UID: "b72ebb8a-18db-4212-8513-59185daa0074"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:21.207951 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.207814 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b72ebb8a-18db-4212-8513-59185daa0074-tls-certs\" (UniqueName: \"kubernetes.io/secret/b72ebb8a-18db-4212-8513-59185daa0074-tls-certs\") pod \"b72ebb8a-18db-4212-8513-59185daa0074\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " Nov 03 09:14:21.207951 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.207859 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-dshm\") pod \"b72ebb8a-18db-4212-8513-59185daa0074\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " Nov 03 09:14:21.217147 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.212676 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72ebb8a-18db-4212-8513-59185daa0074-kube-api-access-bgxgh\" (UniqueName: \"kubernetes.io/projected/b72ebb8a-18db-4212-8513-59185daa0074-kube-api-access-bgxgh\") pod \"b72ebb8a-18db-4212-8513-59185daa0074\" (UID: \"b72ebb8a-18db-4212-8513-59185daa0074\") " Nov 03 09:14:21.217147 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.213307 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:21.217147 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.213328 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:21.217147 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.213342 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:21.222074 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.222034 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72ebb8a-18db-4212-8513-59185daa0074-kube-api-access-bgxgh" pod "b72ebb8a-18db-4212-8513-59185daa0074" (UID: "b72ebb8a-18db-4212-8513-59185daa0074"). InnerVolumeSpecName "kube-api-access-bgxgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:14:21.225668 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.225634 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72ebb8a-18db-4212-8513-59185daa0074-tls-certs" pod "b72ebb8a-18db-4212-8513-59185daa0074" (UID: "b72ebb8a-18db-4212-8513-59185daa0074"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:14:21.225822 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.225735 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-dshm" pod "b72ebb8a-18db-4212-8513-59185daa0074" (UID: "b72ebb8a-18db-4212-8513-59185daa0074"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:21.316025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.315987 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b72ebb8a-18db-4212-8513-59185daa0074-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:21.316025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.316028 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b72ebb8a-18db-4212-8513-59185daa0074-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:21.316314 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.316045 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgxgh\" (UniqueName: \"kubernetes.io/projected/b72ebb8a-18db-4212-8513-59185daa0074-kube-api-access-bgxgh\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:21.686773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.686741 5015 generic.go:356] "Generic (PLEG): container finished" podID="b72ebb8a-18db-4212-8513-59185daa0074" containerID="e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc" exitCode=0 Nov 03 09:14:21.687237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.686798 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" Nov 03 09:14:21.687237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.686815 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" event={"ID":"b72ebb8a-18db-4212-8513-59185daa0074","Type":"ContainerDied","Data":"e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc"} Nov 03 09:14:21.687237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.686852 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn" event={"ID":"b72ebb8a-18db-4212-8513-59185daa0074","Type":"ContainerDied","Data":"bac7190cd09ab19a3d8770dd4a8408619c120f9c7ddb56a2ac4c351998e13e8c"} Nov 03 09:14:21.687237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.686868 5015 scope.go:122] "RemoveContainer" containerID="e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc" Nov 03 09:14:21.689217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.689189 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" event={"ID":"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b","Type":"ContainerStarted","Data":"9810235e28f946dd422a893006a60cd65212c61893614fec96c8aca6151de1db"} Nov 03 09:14:21.699013 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.698993 5015 scope.go:122] "RemoveContainer" containerID="e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc" Nov 03 09:14:21.699094 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.699000 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:14:21.699341 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:14:21.699321 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc\": container with ID starting with e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc not found: ID does not exist" containerID="e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc" Nov 03 09:14:21.699420 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.699347 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc"} err="failed to get container status \"e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc\": rpc error: code = NotFound desc = could not find container \"e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc\": container with ID starting with e20f7fd0ae739e1d01bd9f73eefb6f9ac3293176ff433ad84ab75b542a7156dc not found: ID does not exist" Nov 03 09:14:21.873543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.873499 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn"] Nov 03 09:14:21.878186 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:21.878156 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-04934e/ipp-route-04934e-kserve-7dd67bb465-4qxjn"] Nov 03 09:14:22.261820 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:22.261704 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:14:22.263037 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:22.262981 5015 patch_prober.go:29] interesting pod/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 09:14:22.263158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:22.263061 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 09:14:22.597349 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:22.597264 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:14:22.710218 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:22.710188 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv"] Nov 03 09:14:22.710781 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:22.710734 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" podUID="170b3676-771c-4739-b132-380b3b25972a" containerName="istio-proxy" containerID="cri-o://c1d0d720e0790e9e98bc5a3df2119edef04ca9e6f5b600186623279f8a64ad90" gracePeriod=30 Nov 03 09:14:23.206208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:23.206165 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72ebb8a-18db-4212-8513-59185daa0074" path="/var/lib/kubelet/pods/b72ebb8a-18db-4212-8513-59185daa0074/volumes" Nov 03 09:14:23.261643 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:23.261604 5015 patch_prober.go:29] interesting pod/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 09:14:23.262378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:23.261896 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 09:14:23.427723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:23.427692 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:14:23.427895 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:23.427757 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" podUID="170b3676-771c-4739-b132-380b3b25972a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:14:23.605613 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:23.605585 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:14:24.263072 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:24.263030 5015 patch_prober.go:29] interesting pod/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 09:14:24.263625 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:24.263114 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 09:14:25.263502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:25.262862 5015 patch_prober.go:29] interesting pod/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 09:14:25.263502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:25.262923 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 09:14:26.261949 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:26.261910 5015 patch_prober.go:29] interesting pod/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 09:14:26.265556 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:26.262395 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 09:14:27.262293 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:27.262263 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:14:27.263156 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:27.263134 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:14:27.584859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:27.584778 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:14:27.598849 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:27.598825 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:14:27.984517 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:27.984492 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 09:14:28.042889 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.042859 5015 scope.go:122] "RemoveContainer" containerID="c1d0d720e0790e9e98bc5a3df2119edef04ca9e6f5b600186623279f8a64ad90" Nov 03 09:14:28.124881 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.124745 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-socket\") pod \"170b3676-771c-4739-b132-380b3b25972a\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " Nov 03 09:14:28.124881 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.124826 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-envoy\") pod \"170b3676-771c-4739-b132-380b3b25972a\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " Nov 03 09:14:28.125073 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.124930 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-credential-socket\") pod \"170b3676-771c-4739-b132-380b3b25972a\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " Nov 03 09:14:28.125073 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.124966 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-istio-token\" (UniqueName: \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-istio-token\") pod \"170b3676-771c-4739-b132-380b3b25972a\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " Nov 03 09:14:28.125073 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.125035 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/170b3676-771c-4739-b132-380b3b25972a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170b3676-771c-4739-b132-380b3b25972a-istio-podinfo\") pod \"170b3676-771c-4739-b132-380b3b25972a\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " Nov 03 09:14:28.125183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.125105 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-certs\") pod \"170b3676-771c-4739-b132-380b3b25972a\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " Nov 03 09:14:28.125183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.125128 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-socket" pod "170b3676-771c-4739-b132-380b3b25972a" (UID: "170b3676-771c-4739-b132-380b3b25972a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:28.125183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.125173 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-data\") pod \"170b3676-771c-4739-b132-380b3b25972a\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " Nov 03 09:14:28.125711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.125634 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/170b3676-771c-4739-b132-380b3b25972a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170b3676-771c-4739-b132-380b3b25972a-istiod-ca-cert\") pod \"170b3676-771c-4739-b132-380b3b25972a\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " Nov 03 09:14:28.125826 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.125719 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-kube-api-access-9mklq\" (UniqueName: \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-kube-api-access-9mklq\") pod \"170b3676-771c-4739-b132-380b3b25972a\" (UID: \"170b3676-771c-4739-b132-380b3b25972a\") " Nov 03 09:14:28.125826 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.125740 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-credential-socket" pod "170b3676-771c-4739-b132-380b3b25972a" (UID: "170b3676-771c-4739-b132-380b3b25972a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:28.125950 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.125917 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-data" pod "170b3676-771c-4739-b132-380b3b25972a" (UID: "170b3676-771c-4739-b132-380b3b25972a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:28.126432 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.126402 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-certs" pod "170b3676-771c-4739-b132-380b3b25972a" (UID: "170b3676-771c-4739-b132-380b3b25972a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:28.126791 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.126765 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:28.126887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.126795 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:28.126887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.126810 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:28.126887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.126826 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:28.126887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.126850 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170b3676-771c-4739-b132-380b3b25972a-istiod-ca-cert" pod "170b3676-771c-4739-b132-380b3b25972a" (UID: "170b3676-771c-4739-b132-380b3b25972a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:14:28.128251 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.128166 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-envoy" pod "170b3676-771c-4739-b132-380b3b25972a" (UID: "170b3676-771c-4739-b132-380b3b25972a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:14:28.128350 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.128274 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/170b3676-771c-4739-b132-380b3b25972a-istio-podinfo" pod "170b3676-771c-4739-b132-380b3b25972a" (UID: "170b3676-771c-4739-b132-380b3b25972a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:14:28.128611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.128581 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-istio-token" pod "170b3676-771c-4739-b132-380b3b25972a" (UID: "170b3676-771c-4739-b132-380b3b25972a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:14:28.128975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.128957 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-kube-api-access-9mklq" pod "170b3676-771c-4739-b132-380b3b25972a" (UID: "170b3676-771c-4739-b132-380b3b25972a"). InnerVolumeSpecName "kube-api-access-9mklq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:14:28.228921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.228885 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:28.228921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.228923 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170b3676-771c-4739-b132-380b3b25972a-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:28.229122 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.228938 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170b3676-771c-4739-b132-380b3b25972a-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:28.229122 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.228951 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mklq\" (UniqueName: \"kubernetes.io/projected/170b3676-771c-4739-b132-380b3b25972a-kube-api-access-9mklq\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:28.229122 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.228965 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170b3676-771c-4739-b132-380b3b25972a-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:14:28.751441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.751402 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" event={"ID":"170b3676-771c-4739-b132-380b3b25972a","Type":"ContainerDied","Data":"c1d0d720e0790e9e98bc5a3df2119edef04ca9e6f5b600186623279f8a64ad90"} Nov 03 09:14:28.751920 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.751488 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" event={"ID":"170b3676-771c-4739-b132-380b3b25972a","Type":"ContainerDied","Data":"3fd98063e624af0f34ba80274f6bef52849214663f6cef5c56f94f70ece2c797"} Nov 03 09:14:28.751920 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:28.751497 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv" Nov 03 09:14:29.048991 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:29.048917 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv"] Nov 03 09:14:29.052361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:29.052340 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-qb4xv"] Nov 03 09:14:29.205340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:29.205304 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="170b3676-771c-4739-b132-380b3b25972a" path="/var/lib/kubelet/pods/170b3676-771c-4739-b132-380b3b25972a/volumes" Nov 03 09:14:46.903994 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:14:46.903906 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:14:52.275515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.275455 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn"] Nov 03 09:14:52.276097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.276083 5015 cpu_manager.go:414] "containerMap: removing container" podUID="170b3676-771c-4739-b132-380b3b25972a" containerName="istio-proxy" Nov 03 09:14:52.276143 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.276098 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170b3676-771c-4739-b132-380b3b25972a" containerName="istio-proxy" Nov 03 09:14:52.276143 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.276132 5015 cpu_manager.go:414] "containerMap: removing container" podUID="b72ebb8a-18db-4212-8513-59185daa0074" containerName="main" Nov 03 09:14:52.276143 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.276137 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72ebb8a-18db-4212-8513-59185daa0074" containerName="main" Nov 03 09:14:52.276244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.276234 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="170b3676-771c-4739-b132-380b3b25972a" containerName="istio-proxy" Nov 03 09:14:52.276279 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.276250 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72ebb8a-18db-4212-8513-59185daa0074" containerName="main" Nov 03 09:14:52.279874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.279855 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.346243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.346211 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-0eecfd-openshift-default-dockercfg-8znhp\"" Nov 03 09:14:52.392097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.392057 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn"] Nov 03 09:14:52.437840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.436956 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-socket\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.437840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.437053 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-podinfo\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.437840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.437109 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-envoy\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.437840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.437222 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istiod-ca-cert\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.437840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.437270 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j77q\" (UniqueName: \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-kube-api-access-4j77q\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.437840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.437332 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-data\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.437840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.437389 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-token\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.437840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.437509 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-certs\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.437840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.437547 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-credential-socket\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.538570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.538391 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-token\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.538755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.538573 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-certs\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.538755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.538614 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-credential-socket\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.538755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.538661 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-socket\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.538755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.538700 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-podinfo\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.538755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.538749 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-envoy\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.539069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.538803 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istiod-ca-cert\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.539069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.538848 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j77q\" (UniqueName: \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-kube-api-access-4j77q\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.539069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.538894 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-data\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.539237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.539111 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-credential-socket\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.539426 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.539400 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-data\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.539551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.539413 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-certs\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.540051 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.539712 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-socket\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.540868 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.540844 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istiod-ca-cert\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.565941 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.565904 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-envoy\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.566348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.566132 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-podinfo\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.604510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.602288 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j77q\" (UniqueName: \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-kube-api-access-4j77q\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.608383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.608344 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-token\") pod \"e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:52.893939 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:52.893847 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:53.396700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:53.396672 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn"] Nov 03 09:14:53.398338 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:14:53.398313 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ebc178_9bdf_4b0d_9d34_13b9fc475477.slice/crio-6617fb2ed312cc8dc613f1f9cb9b7da7ffa75ca915574d14f3bb782d2cb7533b WatchSource:0}: Error finding container 6617fb2ed312cc8dc613f1f9cb9b7da7ffa75ca915574d14f3bb782d2cb7533b: Status 404 returned error can't find the container with id 6617fb2ed312cc8dc613f1f9cb9b7da7ffa75ca915574d14f3bb782d2cb7533b Nov 03 09:14:53.400962 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:53.400921 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:14:53.401060 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:53.400995 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:14:53.401060 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:53.401030 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:14:53.952530 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:53.951851 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" event={"ID":"37ebc178-9bdf-4b0d-9d34-13b9fc475477","Type":"ContainerStarted","Data":"601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f"} Nov 03 09:14:53.952530 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:53.951925 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" event={"ID":"37ebc178-9bdf-4b0d-9d34-13b9fc475477","Type":"ContainerStarted","Data":"6617fb2ed312cc8dc613f1f9cb9b7da7ffa75ca915574d14f3bb782d2cb7533b"} Nov 03 09:14:54.125140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:54.125082 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" podStartSLOduration=2.125069459 podStartE2EDuration="2.125069459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:14:54.121302743 +0000 UTC m=+2127.046244978" watchObservedRunningTime="2026-11-03 09:14:54.125069459 +0000 UTC m=+2127.050011676" Nov 03 09:14:54.894585 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:54.894555 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:54.900466 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:54.900440 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:54.958908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:54.958875 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:14:54.959900 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:14:54.959878 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:15:00.171003 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.170963 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc"] Nov 03 09:15:00.175679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.175655 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" Nov 03 09:15:00.176684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.176656 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn"] Nov 03 09:15:00.181920 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.181896 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" Nov 03 09:15:00.189329 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.189306 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766"] Nov 03 09:15:00.193313 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.193296 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" Nov 03 09:15:00.218861 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.218836 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j"] Nov 03 09:15:00.222823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.222805 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv"] Nov 03 09:15:00.222957 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.222940 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" Nov 03 09:15:00.226646 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.226624 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" Nov 03 09:15:00.233773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.233745 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/184e292a-8c9f-42f1-be46-57313bd4e311-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894955-v9xxn\" (UID: \"184e292a-8c9f-42f1-be46-57313bd4e311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" Nov 03 09:15:00.233870 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.233806 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s6ss\" (UniqueName: \"kubernetes.io/projected/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-kube-api-access-4s6ss\") pod \"maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j\" (UID: \"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" Nov 03 09:15:00.233912 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.233901 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j\" (UID: \"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" Nov 03 09:15:00.233956 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.233942 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44fk\" (UniqueName: \"kubernetes.io/projected/455cdb19-5da4-47fd-96be-49c63d856fe8-kube-api-access-g44fk\") pod \"maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc\" (UID: \"455cdb19-5da4-47fd-96be-49c63d856fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" Nov 03 09:15:00.234002 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.233969 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766\" (UID: \"8c093ebc-02f0-4353-84f8-4aca0ceadaa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" Nov 03 09:15:00.234069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.234056 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spz9f\" (UniqueName: \"kubernetes.io/projected/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-kube-api-access-spz9f\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766\" (UID: \"8c093ebc-02f0-4353-84f8-4aca0ceadaa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" Nov 03 09:15:00.234181 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.234162 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q6fl\" (UniqueName: \"kubernetes.io/projected/184e292a-8c9f-42f1-be46-57313bd4e311-kube-api-access-4q6fl\") pod \"maas-api-key-cleanup-29894955-v9xxn\" (UID: \"184e292a-8c9f-42f1-be46-57313bd4e311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" Nov 03 09:15:00.234269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.234252 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455cdb19-5da4-47fd-96be-49c63d856fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc\" (UID: \"455cdb19-5da4-47fd-96be-49c63d856fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" Nov 03 09:15:00.234329 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.234287 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:15:00.269223 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.269190 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc"] Nov 03 09:15:00.274601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.274572 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn"] Nov 03 09:15:00.323246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.321099 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766"] Nov 03 09:15:00.334746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.334701 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j"] Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.335500 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455cdb19-5da4-47fd-96be-49c63d856fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc\" (UID: \"455cdb19-5da4-47fd-96be-49c63d856fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.335540 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrpgb\" (UniqueName: \"kubernetes.io/projected/ce533940-7370-468a-b2b9-48b34440f75c-kube-api-access-zrpgb\") pod \"maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv\" (UID: \"ce533940-7370-468a-b2b9-48b34440f75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.335606 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/184e292a-8c9f-42f1-be46-57313bd4e311-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894955-v9xxn\" (UID: \"184e292a-8c9f-42f1-be46-57313bd4e311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.335650 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s6ss\" (UniqueName: \"kubernetes.io/projected/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-kube-api-access-4s6ss\") pod \"maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j\" (UID: \"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.335714 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j\" (UID: \"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.335764 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44fk\" (UniqueName: \"kubernetes.io/projected/455cdb19-5da4-47fd-96be-49c63d856fe8-kube-api-access-g44fk\") pod \"maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc\" (UID: \"455cdb19-5da4-47fd-96be-49c63d856fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.335792 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766\" (UID: \"8c093ebc-02f0-4353-84f8-4aca0ceadaa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.335832 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spz9f\" (UniqueName: \"kubernetes.io/projected/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-kube-api-access-spz9f\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766\" (UID: \"8c093ebc-02f0-4353-84f8-4aca0ceadaa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.335933 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce533940-7370-468a-b2b9-48b34440f75c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv\" (UID: \"ce533940-7370-468a-b2b9-48b34440f75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.335960 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q6fl\" (UniqueName: \"kubernetes.io/projected/184e292a-8c9f-42f1-be46-57313bd4e311-kube-api-access-4q6fl\") pod \"maas-api-key-cleanup-29894955-v9xxn\" (UID: \"184e292a-8c9f-42f1-be46-57313bd4e311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.337094 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455cdb19-5da4-47fd-96be-49c63d856fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc\" (UID: \"455cdb19-5da4-47fd-96be-49c63d856fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.337159 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/184e292a-8c9f-42f1-be46-57313bd4e311-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894955-v9xxn\" (UID: \"184e292a-8c9f-42f1-be46-57313bd4e311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" Nov 03 09:15:00.337624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.337228 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766\" (UID: \"8c093ebc-02f0-4353-84f8-4aca0ceadaa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" Nov 03 09:15:00.338437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.337677 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j\" (UID: \"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" Nov 03 09:15:00.363026 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.360591 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv"] Nov 03 09:15:00.408522 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.404711 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s6ss\" (UniqueName: \"kubernetes.io/projected/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-kube-api-access-4s6ss\") pod \"maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j\" (UID: \"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" Nov 03 09:15:00.408522 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.405619 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44fk\" (UniqueName: \"kubernetes.io/projected/455cdb19-5da4-47fd-96be-49c63d856fe8-kube-api-access-g44fk\") pod \"maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc\" (UID: \"455cdb19-5da4-47fd-96be-49c63d856fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" Nov 03 09:15:00.418922 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.418881 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spz9f\" (UniqueName: \"kubernetes.io/projected/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-kube-api-access-spz9f\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766\" (UID: \"8c093ebc-02f0-4353-84f8-4aca0ceadaa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" Nov 03 09:15:00.419313 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.419286 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q6fl\" (UniqueName: \"kubernetes.io/projected/184e292a-8c9f-42f1-be46-57313bd4e311-kube-api-access-4q6fl\") pod \"maas-api-key-cleanup-29894955-v9xxn\" (UID: \"184e292a-8c9f-42f1-be46-57313bd4e311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" Nov 03 09:15:00.440465 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.440380 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce533940-7370-468a-b2b9-48b34440f75c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv\" (UID: \"ce533940-7370-468a-b2b9-48b34440f75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" Nov 03 09:15:00.440761 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.440743 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrpgb\" (UniqueName: \"kubernetes.io/projected/ce533940-7370-468a-b2b9-48b34440f75c-kube-api-access-zrpgb\") pod \"maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv\" (UID: \"ce533940-7370-468a-b2b9-48b34440f75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" Nov 03 09:15:00.441182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.441155 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce533940-7370-468a-b2b9-48b34440f75c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv\" (UID: \"ce533940-7370-468a-b2b9-48b34440f75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" Nov 03 09:15:00.494896 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.494862 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" Nov 03 09:15:00.502044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.502013 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrpgb\" (UniqueName: \"kubernetes.io/projected/ce533940-7370-468a-b2b9-48b34440f75c-kube-api-access-zrpgb\") pod \"maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv\" (UID: \"ce533940-7370-468a-b2b9-48b34440f75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" Nov 03 09:15:00.503783 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.503760 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" Nov 03 09:15:00.514532 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.511409 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" Nov 03 09:15:00.546333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.545616 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" Nov 03 09:15:00.546333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.546198 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" Nov 03 09:15:00.806452 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.806277 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v"] Nov 03 09:15:00.841593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.840169 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v"] Nov 03 09:15:00.841593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.840296 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:15:00.895740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.895691 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-0eecfd-dockercfg-7vlhk\"" Nov 03 09:15:00.895740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.895698 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-0eecfd\"" Nov 03 09:15:00.957042 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.956942 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lrs9\" (UniqueName: \"kubernetes.io/projected/2b46ba45-e329-44cd-b50a-cddafe821ba8-kube-api-access-8lrs9\") pod \"payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v\" (UID: \"2b46ba45-e329-44cd-b50a-cddafe821ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:15:00.957185 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.957044 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b46ba45-e329-44cd-b50a-cddafe821ba8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v\" (UID: \"2b46ba45-e329-44cd-b50a-cddafe821ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:15:00.982450 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:00.982416 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv"] Nov 03 09:15:00.983719 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:15:00.983692 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce533940_7370_468a_b2b9_48b34440f75c.slice/crio-b574e897287c6f831475323c5debc56b6129cbf09b09d64611561b159e6db06b WatchSource:0}: Error finding container b574e897287c6f831475323c5debc56b6129cbf09b09d64611561b159e6db06b: Status 404 returned error can't find the container with id b574e897287c6f831475323c5debc56b6129cbf09b09d64611561b159e6db06b Nov 03 09:15:01.006465 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:01.006404 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" event={"ID":"ce533940-7370-468a-b2b9-48b34440f75c","Type":"ContainerStarted","Data":"b574e897287c6f831475323c5debc56b6129cbf09b09d64611561b159e6db06b"} Nov 03 09:15:01.058840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:01.058810 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lrs9\" (UniqueName: \"kubernetes.io/projected/2b46ba45-e329-44cd-b50a-cddafe821ba8-kube-api-access-8lrs9\") pod \"payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v\" (UID: \"2b46ba45-e329-44cd-b50a-cddafe821ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:15:01.058974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:01.058893 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b46ba45-e329-44cd-b50a-cddafe821ba8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v\" (UID: \"2b46ba45-e329-44cd-b50a-cddafe821ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:15:01.059560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:01.059542 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b46ba45-e329-44cd-b50a-cddafe821ba8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v\" (UID: \"2b46ba45-e329-44cd-b50a-cddafe821ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:15:01.120420 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:01.120392 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lrs9\" (UniqueName: \"kubernetes.io/projected/2b46ba45-e329-44cd-b50a-cddafe821ba8-kube-api-access-8lrs9\") pod \"payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v\" (UID: \"2b46ba45-e329-44cd-b50a-cddafe821ba8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:15:01.166310 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:01.166280 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc"] Nov 03 09:15:01.166663 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:15:01.166641 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455cdb19_5da4_47fd_96be_49c63d856fe8.slice/crio-375710ab405b4ad012f064b29f097e30088b8538346e4406d022a3d2093532da WatchSource:0}: Error finding container 375710ab405b4ad012f064b29f097e30088b8538346e4406d022a3d2093532da: Status 404 returned error can't find the container with id 375710ab405b4ad012f064b29f097e30088b8538346e4406d022a3d2093532da Nov 03 09:15:01.169190 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:01.169162 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:15:01.215699 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:15:01.215664 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184e292a_8c9f_42f1_be46_57313bd4e311.slice/crio-8a580997524fcb39b37045d39bcfdf6b8899080b03f1c38edf5b4f715160a11b WatchSource:0}: Error finding container 8a580997524fcb39b37045d39bcfdf6b8899080b03f1c38edf5b4f715160a11b: Status 404 returned error can't find the container with id 8a580997524fcb39b37045d39bcfdf6b8899080b03f1c38edf5b4f715160a11b Nov 03 09:15:01.216287 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:01.216227 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn"] Nov 03 09:15:01.219829 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:15:01.219795 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ac62f3_e0bb_4ea9_9b12_fe81de0600e5.slice/crio-5bee7df87295c1f190f86d42cb96917f751a3963ce3d1be1cde8ecb26ea081b2 WatchSource:0}: Error finding container 5bee7df87295c1f190f86d42cb96917f751a3963ce3d1be1cde8ecb26ea081b2: Status 404 returned error can't find the container with id 5bee7df87295c1f190f86d42cb96917f751a3963ce3d1be1cde8ecb26ea081b2 Nov 03 09:15:01.225445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:01.222799 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j"] Nov 03 09:15:01.225445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:01.225033 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766"] Nov 03 09:15:01.575621 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:01.575591 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v"] Nov 03 09:15:01.641861 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:15:01.641824 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b46ba45_e329_44cd_b50a_cddafe821ba8.slice/crio-81406b754431a3b22be8c0101848a3f55205bf33ccb2a01eaf9b54757d8d0ea9 WatchSource:0}: Error finding container 81406b754431a3b22be8c0101848a3f55205bf33ccb2a01eaf9b54757d8d0ea9: Status 404 returned error can't find the container with id 81406b754431a3b22be8c0101848a3f55205bf33ccb2a01eaf9b54757d8d0ea9 Nov 03 09:15:02.019358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.019318 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" event={"ID":"2b46ba45-e329-44cd-b50a-cddafe821ba8","Type":"ContainerStarted","Data":"648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81"} Nov 03 09:15:02.019583 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.019389 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" event={"ID":"2b46ba45-e329-44cd-b50a-cddafe821ba8","Type":"ContainerStarted","Data":"81406b754431a3b22be8c0101848a3f55205bf33ccb2a01eaf9b54757d8d0ea9"} Nov 03 09:15:02.019583 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.019572 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:15:02.021412 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.021386 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" event={"ID":"8c093ebc-02f0-4353-84f8-4aca0ceadaa6","Type":"ContainerStarted","Data":"045b4809fed20a8a2f2bf2a25c566fee57fbaf1196bfacbf1f7755eaf96bc213"} Nov 03 09:15:02.021601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.021436 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" event={"ID":"8c093ebc-02f0-4353-84f8-4aca0ceadaa6","Type":"ContainerStarted","Data":"df4984f055f596abc2d4b657c7b4d95f68800e43b5e3e18820e8ebde4efe60f8"} Nov 03 09:15:02.023189 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.023164 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" event={"ID":"184e292a-8c9f-42f1-be46-57313bd4e311","Type":"ContainerStarted","Data":"e30be9690f70d77cb717788bbe8bb6a00f8c9302db53c6c1696c2f1121209d2a"} Nov 03 09:15:02.023317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.023210 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" event={"ID":"184e292a-8c9f-42f1-be46-57313bd4e311","Type":"ContainerStarted","Data":"8a580997524fcb39b37045d39bcfdf6b8899080b03f1c38edf5b4f715160a11b"} Nov 03 09:15:02.024867 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.024840 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" event={"ID":"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5","Type":"ContainerStarted","Data":"4f0929b3489909cc5f536481fa589dc4c0b27f2c51037fb538e07ed0845c453b"} Nov 03 09:15:02.024981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.024888 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" event={"ID":"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5","Type":"ContainerStarted","Data":"5bee7df87295c1f190f86d42cb96917f751a3963ce3d1be1cde8ecb26ea081b2"} Nov 03 09:15:02.026523 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.026501 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" event={"ID":"455cdb19-5da4-47fd-96be-49c63d856fe8","Type":"ContainerStarted","Data":"2341d9bd89696cd1fe6487b46734636f1fb147c143f1170159debe6c1f6ec4ab"} Nov 03 09:15:02.026627 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.026547 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" event={"ID":"455cdb19-5da4-47fd-96be-49c63d856fe8","Type":"ContainerStarted","Data":"375710ab405b4ad012f064b29f097e30088b8538346e4406d022a3d2093532da"} Nov 03 09:15:02.028071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.028048 5015 generic.go:356] "Generic (PLEG): container finished" podID="ce533940-7370-468a-b2b9-48b34440f75c" containerID="3bedf4850bcdecbc88b8210ea9ed9d704270c85f20244d8e60cb8df9e5fac832" exitCode=60 Nov 03 09:15:02.028185 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.028086 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" event={"ID":"ce533940-7370-468a-b2b9-48b34440f75c","Type":"ContainerDied","Data":"3bedf4850bcdecbc88b8210ea9ed9d704270c85f20244d8e60cb8df9e5fac832"} Nov 03 09:15:02.028326 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.028310 5015 scope.go:122] "RemoveContainer" containerID="3bedf4850bcdecbc88b8210ea9ed9d704270c85f20244d8e60cb8df9e5fac832" Nov 03 09:15:02.136853 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.136789 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" podStartSLOduration=2.136768624 podStartE2EDuration="2.136768624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:02.135573735 +0000 UTC m=+2135.060515953" watchObservedRunningTime="2026-11-03 09:15:02.136768624 +0000 UTC m=+2135.061710842" Nov 03 09:15:02.262623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.261364 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" podStartSLOduration=2.261344355 podStartE2EDuration="2.261344355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:02.256686051 +0000 UTC m=+2135.181628269" watchObservedRunningTime="2026-11-03 09:15:02.261344355 +0000 UTC m=+2135.186286570" Nov 03 09:15:02.375365 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.375315 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" podStartSLOduration=2.37530064 podStartE2EDuration="2.37530064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:02.371269683 +0000 UTC m=+2135.296211902" watchObservedRunningTime="2026-11-03 09:15:02.37530064 +0000 UTC m=+2135.300242859" Nov 03 09:15:02.487901 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.487842 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" podStartSLOduration=2.487828212 podStartE2EDuration="2.487828212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:02.486434685 +0000 UTC m=+2135.411376921" watchObservedRunningTime="2026-11-03 09:15:02.487828212 +0000 UTC m=+2135.412770429" Nov 03 09:15:02.655189 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.655154 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw"] Nov 03 09:15:02.659891 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.659865 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.672506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.672461 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx"] Nov 03 09:15:02.676684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.676666 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.715229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.715199 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-26e861-openshift-default-dockercfg-4wl7s\"" Nov 03 09:15:02.719860 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.719829 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw"] Nov 03 09:15:02.725196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.725135 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" podStartSLOduration=2.725116211 podStartE2EDuration="2.725116211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:02.721027426 +0000 UTC m=+2135.645969643" watchObservedRunningTime="2026-11-03 09:15:02.725116211 +0000 UTC m=+2135.650058431" Nov 03 09:15:02.745992 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.745954 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx"] Nov 03 09:15:02.793739 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.793641 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqhp\" (UniqueName: \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-kube-api-access-rnqhp\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.793739 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.793694 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.793943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.793763 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c41adb7-c695-4531-8811-86e0c73858da-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.793943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.793841 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.793943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.793885 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.794103 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.793947 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.794103 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.793974 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.794103 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.794041 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.794103 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.794092 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.794303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.794262 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.794359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.794319 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.794433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.794415 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.794590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.794572 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c41adb7-c695-4531-8811-86e0c73858da-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.794670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.794613 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.794670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.794637 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjld\" (UniqueName: \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-kube-api-access-scjld\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.794763 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.794742 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.794805 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.794775 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.794805 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.794797 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.895986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.895949 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.895986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.895999 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.896199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896028 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.896199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896057 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.896199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896090 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.896199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896135 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.896414 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896212 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.896414 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896256 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.896414 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896307 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.896414 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896357 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c41adb7-c695-4531-8811-86e0c73858da-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.896414 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896404 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.896703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896434 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjld\" (UniqueName: \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-kube-api-access-scjld\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.896703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896488 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.896703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896516 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.896703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896550 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.896703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896577 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqhp\" (UniqueName: \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-kube-api-access-rnqhp\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.896703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896589 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.896703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896644 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.897112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896737 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.897112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896859 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.897112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896971 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.897112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.897052 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.897112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.897096 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.897391 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.897177 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.897391 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.896599 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.897391 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.897249 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c41adb7-c695-4531-8811-86e0c73858da-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.897391 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.897303 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.898396 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.898349 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c41adb7-c695-4531-8811-86e0c73858da-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.901588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.901213 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.901588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.901274 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c41adb7-c695-4531-8811-86e0c73858da-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.901588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.901432 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.902276 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.902235 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.956525 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.956437 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:02.956525 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.956517 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.956947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.956924 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjld\" (UniqueName: \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-kube-api-access-scjld\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:02.956947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:02.956941 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqhp\" (UniqueName: \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-kube-api-access-rnqhp\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:03.010462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.010429 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:03.017840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.017817 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:03.045821 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.045736 5015 generic.go:356] "Generic (PLEG): container finished" podID="184e292a-8c9f-42f1-be46-57313bd4e311" containerID="e30be9690f70d77cb717788bbe8bb6a00f8c9302db53c6c1696c2f1121209d2a" exitCode=60 Nov 03 09:15:03.045973 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.045816 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" event={"ID":"184e292a-8c9f-42f1-be46-57313bd4e311","Type":"ContainerDied","Data":"e30be9690f70d77cb717788bbe8bb6a00f8c9302db53c6c1696c2f1121209d2a"} Nov 03 09:15:03.048095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.048043 5015 generic.go:356] "Generic (PLEG): container finished" podID="f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" containerID="4f0929b3489909cc5f536481fa589dc4c0b27f2c51037fb538e07ed0845c453b" exitCode=60 Nov 03 09:15:03.048959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.048114 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" event={"ID":"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5","Type":"ContainerDied","Data":"4f0929b3489909cc5f536481fa589dc4c0b27f2c51037fb538e07ed0845c453b"} Nov 03 09:15:03.048959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.048135 5015 scope.go:122] "RemoveContainer" containerID="e30be9690f70d77cb717788bbe8bb6a00f8c9302db53c6c1696c2f1121209d2a" Nov 03 09:15:03.048959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.048649 5015 scope.go:122] "RemoveContainer" containerID="4f0929b3489909cc5f536481fa589dc4c0b27f2c51037fb538e07ed0845c453b" Nov 03 09:15:03.052768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.052704 5015 generic.go:356] "Generic (PLEG): container finished" podID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerID="2341d9bd89696cd1fe6487b46734636f1fb147c143f1170159debe6c1f6ec4ab" exitCode=60 Nov 03 09:15:03.052835 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.052806 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" event={"ID":"455cdb19-5da4-47fd-96be-49c63d856fe8","Type":"ContainerDied","Data":"2341d9bd89696cd1fe6487b46734636f1fb147c143f1170159debe6c1f6ec4ab"} Nov 03 09:15:03.054199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.053219 5015 scope.go:122] "RemoveContainer" containerID="2341d9bd89696cd1fe6487b46734636f1fb147c143f1170159debe6c1f6ec4ab" Nov 03 09:15:03.056027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.055371 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" event={"ID":"ce533940-7370-468a-b2b9-48b34440f75c","Type":"ContainerStarted","Data":"8bd0ffcc366baeebd072630173af504f0a6a2b53e341b3f6ae445051574abf3b"} Nov 03 09:15:03.059245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.058030 5015 generic.go:356] "Generic (PLEG): container finished" podID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerID="045b4809fed20a8a2f2bf2a25c566fee57fbaf1196bfacbf1f7755eaf96bc213" exitCode=60 Nov 03 09:15:03.059245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.059041 5015 scope.go:122] "RemoveContainer" containerID="045b4809fed20a8a2f2bf2a25c566fee57fbaf1196bfacbf1f7755eaf96bc213" Nov 03 09:15:03.059245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.059058 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" event={"ID":"8c093ebc-02f0-4353-84f8-4aca0ceadaa6","Type":"ContainerDied","Data":"045b4809fed20a8a2f2bf2a25c566fee57fbaf1196bfacbf1f7755eaf96bc213"} Nov 03 09:15:03.427664 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.427183 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw"] Nov 03 09:15:03.450895 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.450859 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:15:03.451100 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.451084 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:15:03.451247 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.451232 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:15:03.471890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.471857 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx"] Nov 03 09:15:03.521081 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.521022 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" podStartSLOduration=3.521010252 podStartE2EDuration="3.521010252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:03.520112678 +0000 UTC m=+2136.445054893" watchObservedRunningTime="2026-11-03 09:15:03.521010252 +0000 UTC m=+2136.445952469" Nov 03 09:15:03.542525 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:15:03.539925 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c41adb7_c695_4531_8811_86e0c73858da.slice/crio-70955dbe94aafd6d859fefa85bac11d946a904d92871e786fed451ade78396cd WatchSource:0}: Error finding container 70955dbe94aafd6d859fefa85bac11d946a904d92871e786fed451ade78396cd: Status 404 returned error can't find the container with id 70955dbe94aafd6d859fefa85bac11d946a904d92871e786fed451ade78396cd Nov 03 09:15:03.551398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.551352 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:15:03.551533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.551453 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:15:03.551533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:03.551521 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:15:04.071079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.071037 5015 generic.go:356] "Generic (PLEG): container finished" podID="ce533940-7370-468a-b2b9-48b34440f75c" containerID="8bd0ffcc366baeebd072630173af504f0a6a2b53e341b3f6ae445051574abf3b" exitCode=60 Nov 03 09:15:04.072686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.072055 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" event={"ID":"ce533940-7370-468a-b2b9-48b34440f75c","Type":"ContainerDied","Data":"8bd0ffcc366baeebd072630173af504f0a6a2b53e341b3f6ae445051574abf3b"} Nov 03 09:15:04.072686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.071827 5015 scope.go:122] "RemoveContainer" containerID="8bd0ffcc366baeebd072630173af504f0a6a2b53e341b3f6ae445051574abf3b" Nov 03 09:15:04.072686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.072128 5015 scope.go:122] "RemoveContainer" containerID="3bedf4850bcdecbc88b8210ea9ed9d704270c85f20244d8e60cb8df9e5fac832" Nov 03 09:15:04.073263 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:04.073182 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv_odh-ai-gateway-infra(ce533940-7370-468a-b2b9-48b34440f75c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" podUID="ce533940-7370-468a-b2b9-48b34440f75c" Nov 03 09:15:04.077802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.077672 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" event={"ID":"8c093ebc-02f0-4353-84f8-4aca0ceadaa6","Type":"ContainerStarted","Data":"d0c7901caa1bd2423701d17dcb4a30c5a6f2ad57c7f56768043851e7e7f8f155"} Nov 03 09:15:04.082053 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.082030 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" event={"ID":"9c41adb7-c695-4531-8811-86e0c73858da","Type":"ContainerStarted","Data":"d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571"} Nov 03 09:15:04.082216 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.082078 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" event={"ID":"9c41adb7-c695-4531-8811-86e0c73858da","Type":"ContainerStarted","Data":"70955dbe94aafd6d859fefa85bac11d946a904d92871e786fed451ade78396cd"} Nov 03 09:15:04.084561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.084537 5015 generic.go:356] "Generic (PLEG): container finished" podID="184e292a-8c9f-42f1-be46-57313bd4e311" containerID="8ac0f68a6c5676e96e3b87b0db70de17bf62afd79effdfa129afe4dda056029e" exitCode=60 Nov 03 09:15:04.084665 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.084620 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" event={"ID":"184e292a-8c9f-42f1-be46-57313bd4e311","Type":"ContainerDied","Data":"8ac0f68a6c5676e96e3b87b0db70de17bf62afd79effdfa129afe4dda056029e"} Nov 03 09:15:04.084914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.084898 5015 scope.go:122] "RemoveContainer" containerID="8ac0f68a6c5676e96e3b87b0db70de17bf62afd79effdfa129afe4dda056029e" Nov 03 09:15:04.085187 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:04.085165 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894955-v9xxn_odh-ai-gateway-infra(184e292a-8c9f-42f1-be46-57313bd4e311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" Nov 03 09:15:04.086858 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.086836 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" event={"ID":"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5","Type":"ContainerStarted","Data":"7786763dc0d6d4816abf086ee88314975fe48888ec4b48842e8758dd3b993f80"} Nov 03 09:15:04.089221 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.089194 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" event={"ID":"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa","Type":"ContainerStarted","Data":"291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8"} Nov 03 09:15:04.089364 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.089241 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" event={"ID":"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa","Type":"ContainerStarted","Data":"c52a716360b53e8a7db727f83c68d3b633ab193a5cf857ad3c727a6de9631863"} Nov 03 09:15:04.090746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.090729 5015 scope.go:122] "RemoveContainer" containerID="e30be9690f70d77cb717788bbe8bb6a00f8c9302db53c6c1696c2f1121209d2a" Nov 03 09:15:04.091282 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.091258 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" event={"ID":"455cdb19-5da4-47fd-96be-49c63d856fe8","Type":"ContainerStarted","Data":"d3ce0934e793aef2e5dee472a3397c007a930a9e8300efe9b211d614a3d8e3a6"} Nov 03 09:15:04.599924 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.599819 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" podStartSLOduration=2.599803554 podStartE2EDuration="2.599803554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:04.598164319 +0000 UTC m=+2137.523106539" watchObservedRunningTime="2026-11-03 09:15:04.599803554 +0000 UTC m=+2137.524745770" Nov 03 09:15:04.859560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:04.859347 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" podStartSLOduration=2.859325423 podStartE2EDuration="2.859325423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:04.85601638 +0000 UTC m=+2137.780958601" watchObservedRunningTime="2026-11-03 09:15:04.859325423 +0000 UTC m=+2137.784267643" Nov 03 09:15:05.010670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.010641 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:05.012204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.012176 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 09:15:05.012350 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.012230 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 09:15:05.018404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.018378 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:05.019723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.019700 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 09:15:05.019814 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.019748 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" podUID="9c41adb7-c695-4531-8811-86e0c73858da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 09:15:05.102995 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.102963 5015 scope.go:122] "RemoveContainer" containerID="8ac0f68a6c5676e96e3b87b0db70de17bf62afd79effdfa129afe4dda056029e" Nov 03 09:15:05.103550 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:05.103526 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894955-v9xxn_odh-ai-gateway-infra(184e292a-8c9f-42f1-be46-57313bd4e311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" Nov 03 09:15:05.104088 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.104067 5015 generic.go:356] "Generic (PLEG): container finished" podID="f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" containerID="7786763dc0d6d4816abf086ee88314975fe48888ec4b48842e8758dd3b993f80" exitCode=60 Nov 03 09:15:05.104222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.104204 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" event={"ID":"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5","Type":"ContainerDied","Data":"7786763dc0d6d4816abf086ee88314975fe48888ec4b48842e8758dd3b993f80"} Nov 03 09:15:05.104273 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.104259 5015 scope.go:122] "RemoveContainer" containerID="4f0929b3489909cc5f536481fa589dc4c0b27f2c51037fb538e07ed0845c453b" Nov 03 09:15:05.104358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.104343 5015 scope.go:122] "RemoveContainer" containerID="7786763dc0d6d4816abf086ee88314975fe48888ec4b48842e8758dd3b993f80" Nov 03 09:15:05.104603 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:05.104586 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j_odh-ai-gateway-infra(f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" podUID="f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" Nov 03 09:15:05.107082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.107061 5015 generic.go:356] "Generic (PLEG): container finished" podID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerID="d3ce0934e793aef2e5dee472a3397c007a930a9e8300efe9b211d614a3d8e3a6" exitCode=60 Nov 03 09:15:05.107146 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.107098 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" event={"ID":"455cdb19-5da4-47fd-96be-49c63d856fe8","Type":"ContainerDied","Data":"d3ce0934e793aef2e5dee472a3397c007a930a9e8300efe9b211d614a3d8e3a6"} Nov 03 09:15:05.107327 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.107314 5015 scope.go:122] "RemoveContainer" containerID="d3ce0934e793aef2e5dee472a3397c007a930a9e8300efe9b211d614a3d8e3a6" Nov 03 09:15:05.107602 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:05.107582 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc_odh-ai-gateway-infra(455cdb19-5da4-47fd-96be-49c63d856fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" Nov 03 09:15:05.108999 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.108981 5015 scope.go:122] "RemoveContainer" containerID="8bd0ffcc366baeebd072630173af504f0a6a2b53e341b3f6ae445051574abf3b" Nov 03 09:15:05.109205 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:05.109183 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv_odh-ai-gateway-infra(ce533940-7370-468a-b2b9-48b34440f75c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" podUID="ce533940-7370-468a-b2b9-48b34440f75c" Nov 03 09:15:05.110627 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.110574 5015 generic.go:356] "Generic (PLEG): container finished" podID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerID="d0c7901caa1bd2423701d17dcb4a30c5a6f2ad57c7f56768043851e7e7f8f155" exitCode=60 Nov 03 09:15:05.110724 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.110647 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" event={"ID":"8c093ebc-02f0-4353-84f8-4aca0ceadaa6","Type":"ContainerDied","Data":"d0c7901caa1bd2423701d17dcb4a30c5a6f2ad57c7f56768043851e7e7f8f155"} Nov 03 09:15:05.110865 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.110847 5015 scope.go:122] "RemoveContainer" containerID="d0c7901caa1bd2423701d17dcb4a30c5a6f2ad57c7f56768043851e7e7f8f155" Nov 03 09:15:05.111103 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:05.111081 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766_odh-ai-gateway-infra(8c093ebc-02f0-4353-84f8-4aca0ceadaa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" Nov 03 09:15:05.117982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.117961 5015 scope.go:122] "RemoveContainer" containerID="2341d9bd89696cd1fe6487b46734636f1fb147c143f1170159debe6c1f6ec4ab" Nov 03 09:15:05.131221 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:05.131200 5015 scope.go:122] "RemoveContainer" containerID="045b4809fed20a8a2f2bf2a25c566fee57fbaf1196bfacbf1f7755eaf96bc213" Nov 03 09:15:06.010746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:06.010717 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 09:15:06.011182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:06.011158 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 09:15:06.018131 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:06.018105 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 09:15:06.018207 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:06.018158 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" podUID="9c41adb7-c695-4531-8811-86e0c73858da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 09:15:06.119855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:06.119823 5015 scope.go:122] "RemoveContainer" containerID="d3ce0934e793aef2e5dee472a3397c007a930a9e8300efe9b211d614a3d8e3a6" Nov 03 09:15:06.120188 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:06.120160 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc_odh-ai-gateway-infra(455cdb19-5da4-47fd-96be-49c63d856fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" Nov 03 09:15:06.121333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:06.121315 5015 scope.go:122] "RemoveContainer" containerID="d0c7901caa1bd2423701d17dcb4a30c5a6f2ad57c7f56768043851e7e7f8f155" Nov 03 09:15:06.121599 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:06.121578 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766_odh-ai-gateway-infra(8c093ebc-02f0-4353-84f8-4aca0ceadaa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" Nov 03 09:15:06.123071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:06.123049 5015 scope.go:122] "RemoveContainer" containerID="7786763dc0d6d4816abf086ee88314975fe48888ec4b48842e8758dd3b993f80" Nov 03 09:15:06.123539 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:06.123465 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j_odh-ai-gateway-infra(f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" podUID="f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" Nov 03 09:15:07.010734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:07.010695 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 09:15:07.010941 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:07.010753 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 09:15:07.018655 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:07.018631 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 09:15:07.018768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:07.018680 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" podUID="9c41adb7-c695-4531-8811-86e0c73858da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 09:15:08.012262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:08.012222 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 09:15:08.012668 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:08.012274 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 09:15:08.019024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:08.018995 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 09:15:08.019118 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:08.019049 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" podUID="9c41adb7-c695-4531-8811-86e0c73858da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 09:15:09.011506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:09.010868 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 09:15:09.011506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:09.010930 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 09:15:09.020155 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:09.020113 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 09:15:09.020626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:09.020175 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" podUID="9c41adb7-c695-4531-8811-86e0c73858da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 09:15:10.011278 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.011241 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 09:15:10.011614 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.011576 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 09:15:10.018642 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.018612 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 09:15:10.018793 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.018667 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" podUID="9c41adb7-c695-4531-8811-86e0c73858da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 09:15:10.267676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.267587 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j"] Nov 03 09:15:10.538115 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.537133 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" Nov 03 09:15:10.583156 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.583122 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl"] Nov 03 09:15:10.583492 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.583427 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" podUID="24a0503c-ea10-46b4-91cc-665171c35c2c" containerName="payload-processing" containerID="cri-o://e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e" gracePeriod=30 Nov 03 09:15:10.658250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.658214 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-maas-api-service-ca\") pod \"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5\" (UID: \"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5\") " Nov 03 09:15:10.658470 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.658328 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-kube-api-access-4s6ss\" (UniqueName: \"kubernetes.io/projected/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-kube-api-access-4s6ss\") pod \"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5\" (UID: \"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5\") " Nov 03 09:15:10.658723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.658685 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-maas-api-service-ca" pod "f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" (UID: "f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:10.658950 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.658930 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:10.661699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.661665 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-kube-api-access-4s6ss" pod "f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" (UID: "f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5"). InnerVolumeSpecName "kube-api-access-4s6ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:10.693618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.692706 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb"] Nov 03 09:15:10.693618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.692974 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" podUID="569b803e-986a-47b3-8552-3f80dd115e23" containerName="payload-pre-processing" containerID="cri-o://060e445235ad1f1486f44ebca35574b048566cd92ba64c63e3b773514f15e412" gracePeriod=30 Nov 03 09:15:10.760817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.760730 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s6ss\" (UniqueName: \"kubernetes.io/projected/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5-kube-api-access-4s6ss\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:10.986101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:10.985850 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:15:11.014121 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.013448 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 09:15:11.014121 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.013547 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 09:15:11.018250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.018219 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 09:15:11.018395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.018276 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" podUID="9c41adb7-c695-4531-8811-86e0c73858da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 09:15:11.029645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.029401 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx"] Nov 03 09:15:11.066101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.066025 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a0503c-ea10-46b4-91cc-665171c35c2c-kube-api-access-kxqq9\" (UniqueName: \"kubernetes.io/projected/24a0503c-ea10-46b4-91cc-665171c35c2c-kube-api-access-kxqq9\") pod \"24a0503c-ea10-46b4-91cc-665171c35c2c\" (UID: \"24a0503c-ea10-46b4-91cc-665171c35c2c\") " Nov 03 09:15:11.066258 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.066241 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a0503c-ea10-46b4-91cc-665171c35c2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a0503c-ea10-46b4-91cc-665171c35c2c-plugins-config-volume\") pod \"24a0503c-ea10-46b4-91cc-665171c35c2c\" (UID: \"24a0503c-ea10-46b4-91cc-665171c35c2c\") " Nov 03 09:15:11.067021 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.066988 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a0503c-ea10-46b4-91cc-665171c35c2c-plugins-config-volume" pod "24a0503c-ea10-46b4-91cc-665171c35c2c" (UID: "24a0503c-ea10-46b4-91cc-665171c35c2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:11.070437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.070406 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a0503c-ea10-46b4-91cc-665171c35c2c-kube-api-access-kxqq9" pod "24a0503c-ea10-46b4-91cc-665171c35c2c" (UID: "24a0503c-ea10-46b4-91cc-665171c35c2c"). InnerVolumeSpecName "kube-api-access-kxqq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:11.174520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.167918 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxqq9\" (UniqueName: \"kubernetes.io/projected/24a0503c-ea10-46b4-91cc-665171c35c2c-kube-api-access-kxqq9\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:11.174520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.167955 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a0503c-ea10-46b4-91cc-665171c35c2c-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:11.274927 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.274894 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" Nov 03 09:15:11.280525 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.278431 5015 generic.go:356] "Generic (PLEG): container finished" podID="24a0503c-ea10-46b4-91cc-665171c35c2c" containerID="e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e" exitCode=0 Nov 03 09:15:11.280525 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.278587 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" Nov 03 09:15:11.342565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.340097 5015 generic.go:356] "Generic (PLEG): container finished" podID="569b803e-986a-47b3-8552-3f80dd115e23" containerID="060e445235ad1f1486f44ebca35574b048566cd92ba64c63e3b773514f15e412" exitCode=0 Nov 03 09:15:11.533373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.533339 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j" event={"ID":"f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5","Type":"ContainerDied","Data":"5bee7df87295c1f190f86d42cb96917f751a3963ce3d1be1cde8ecb26ea081b2"} Nov 03 09:15:11.533677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.533646 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7"] Nov 03 09:15:11.533868 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.533848 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" event={"ID":"24a0503c-ea10-46b4-91cc-665171c35c2c","Type":"ContainerDied","Data":"e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e"} Nov 03 09:15:11.533999 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.533982 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl" event={"ID":"24a0503c-ea10-46b4-91cc-665171c35c2c","Type":"ContainerDied","Data":"f87a9ddde3c2c3adce79f50e8542f0181a1826d3843cd633e66794910e68d311"} Nov 03 09:15:11.534093 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.534079 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" event={"ID":"569b803e-986a-47b3-8552-3f80dd115e23","Type":"ContainerDied","Data":"060e445235ad1f1486f44ebca35574b048566cd92ba64c63e3b773514f15e412"} Nov 03 09:15:11.534197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.533877 5015 scope.go:122] "RemoveContainer" containerID="7786763dc0d6d4816abf086ee88314975fe48888ec4b48842e8758dd3b993f80" Nov 03 09:15:11.612106 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.612025 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl"] Nov 03 09:15:11.613461 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.613433 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c78f2b-649b9b9b4d-v48sl"] Nov 03 09:15:11.642729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.642622 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:15:11.644250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.644159 5015 scope.go:122] "RemoveContainer" containerID="e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e" Nov 03 09:15:11.664012 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.663468 5015 scope.go:122] "RemoveContainer" containerID="e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e" Nov 03 09:15:11.664012 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:11.663856 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e\": container with ID starting with e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e not found: ID does not exist" containerID="e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e" Nov 03 09:15:11.664012 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.663895 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e"} err="failed to get container status \"e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e\": rpc error: code = NotFound desc = could not find container \"e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e\": container with ID starting with e7c9b14a185aa360c8db3a209956e53ee087c7406a813c6d583cdc0561ef706e not found: ID does not exist" Nov 03 09:15:11.831164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.831115 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569b803e-986a-47b3-8552-3f80dd115e23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569b803e-986a-47b3-8552-3f80dd115e23-plugins-config-volume\") pod \"569b803e-986a-47b3-8552-3f80dd115e23\" (UID: \"569b803e-986a-47b3-8552-3f80dd115e23\") " Nov 03 09:15:11.831443 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.831422 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569b803e-986a-47b3-8552-3f80dd115e23-kube-api-access-gb477\" (UniqueName: \"kubernetes.io/projected/569b803e-986a-47b3-8552-3f80dd115e23-kube-api-access-gb477\") pod \"569b803e-986a-47b3-8552-3f80dd115e23\" (UID: \"569b803e-986a-47b3-8552-3f80dd115e23\") " Nov 03 09:15:11.831831 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.831800 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569b803e-986a-47b3-8552-3f80dd115e23-plugins-config-volume" pod "569b803e-986a-47b3-8552-3f80dd115e23" (UID: "569b803e-986a-47b3-8552-3f80dd115e23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:11.832106 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.832084 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569b803e-986a-47b3-8552-3f80dd115e23-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:11.834108 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.834080 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j"] Nov 03 09:15:11.840499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.840201 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c78f2b-29894955-mf87j"] Nov 03 09:15:11.849906 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.848933 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569b803e-986a-47b3-8552-3f80dd115e23-kube-api-access-gb477" pod "569b803e-986a-47b3-8552-3f80dd115e23" (UID: "569b803e-986a-47b3-8552-3f80dd115e23"). InnerVolumeSpecName "kube-api-access-gb477". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:11.933129 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:11.933047 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb477\" (UniqueName: \"kubernetes.io/projected/569b803e-986a-47b3-8552-3f80dd115e23-kube-api-access-gb477\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:12.059365 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:12.059326 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:12.059935 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:12.059906 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:12.060838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:12.060810 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:15:12.125378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:12.125330 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:12.125755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:12.125732 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:12.127234 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:12.127213 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:15:12.358869 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:12.358842 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" Nov 03 09:15:12.359345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:12.358983 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb" event={"ID":"569b803e-986a-47b3-8552-3f80dd115e23","Type":"ContainerDied","Data":"b24029ca7d69da77dd774c84f79ca4b029fbdd57cec3dbb13e82ebce05373572"} Nov 03 09:15:12.359345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:12.359051 5015 scope.go:122] "RemoveContainer" containerID="060e445235ad1f1486f44ebca35574b048566cd92ba64c63e3b773514f15e412" Nov 03 09:15:12.557975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:12.546947 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb"] Nov 03 09:15:12.557975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:12.557847 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c78f2b-5b9f548df5-qcwbb"] Nov 03 09:15:13.061748 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:13.061719 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:15:13.211543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:13.211508 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a0503c-ea10-46b4-91cc-665171c35c2c" path="/var/lib/kubelet/pods/24a0503c-ea10-46b4-91cc-665171c35c2c/volumes" Nov 03 09:15:13.212627 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:13.212584 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569b803e-986a-47b3-8552-3f80dd115e23" path="/var/lib/kubelet/pods/569b803e-986a-47b3-8552-3f80dd115e23/volumes" Nov 03 09:15:13.213260 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:13.213240 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" path="/var/lib/kubelet/pods/f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5/volumes" Nov 03 09:15:16.030577 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.030519 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" podUID="28e9f4e5-1c70-40c9-a56d-dff029b79b11" containerName="payload-processing" containerID="cri-o://7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76" gracePeriod=25 Nov 03 09:15:16.197182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.197150 5015 scope.go:122] "RemoveContainer" containerID="d0c7901caa1bd2423701d17dcb4a30c5a6f2ad57c7f56768043851e7e7f8f155" Nov 03 09:15:16.197378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.197299 5015 scope.go:122] "RemoveContainer" containerID="d3ce0934e793aef2e5dee472a3397c007a930a9e8300efe9b211d614a3d8e3a6" Nov 03 09:15:16.197439 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.197403 5015 scope.go:122] "RemoveContainer" containerID="8bd0ffcc366baeebd072630173af504f0a6a2b53e341b3f6ae445051574abf3b" Nov 03 09:15:16.345658 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.345248 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:15:16.400979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.400945 5015 generic.go:356] "Generic (PLEG): container finished" podID="28e9f4e5-1c70-40c9-a56d-dff029b79b11" containerID="7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76" exitCode=0 Nov 03 09:15:16.401151 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.400995 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" Nov 03 09:15:16.401151 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.401035 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" event={"ID":"28e9f4e5-1c70-40c9-a56d-dff029b79b11","Type":"ContainerDied","Data":"7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76"} Nov 03 09:15:16.401151 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.401092 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx" event={"ID":"28e9f4e5-1c70-40c9-a56d-dff029b79b11","Type":"ContainerDied","Data":"9ec423fb0b8cbc0be567cbff0bae37f5227a788dfc3af0fc6b7e9cd93134462d"} Nov 03 09:15:16.401151 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.401117 5015 scope.go:122] "RemoveContainer" containerID="7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76" Nov 03 09:15:16.406574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.406551 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e9f4e5-1c70-40c9-a56d-dff029b79b11-kube-api-access-kq9pw\" (UniqueName: \"kubernetes.io/projected/28e9f4e5-1c70-40c9-a56d-dff029b79b11-kube-api-access-kq9pw\") pod \"28e9f4e5-1c70-40c9-a56d-dff029b79b11\" (UID: \"28e9f4e5-1c70-40c9-a56d-dff029b79b11\") " Nov 03 09:15:16.406688 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.406666 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e9f4e5-1c70-40c9-a56d-dff029b79b11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e9f4e5-1c70-40c9-a56d-dff029b79b11-plugins-config-volume\") pod \"28e9f4e5-1c70-40c9-a56d-dff029b79b11\" (UID: \"28e9f4e5-1c70-40c9-a56d-dff029b79b11\") " Nov 03 09:15:16.407020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.407000 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e9f4e5-1c70-40c9-a56d-dff029b79b11-plugins-config-volume" pod "28e9f4e5-1c70-40c9-a56d-dff029b79b11" (UID: "28e9f4e5-1c70-40c9-a56d-dff029b79b11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:16.407282 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.407267 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e9f4e5-1c70-40c9-a56d-dff029b79b11-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:16.416001 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.415982 5015 scope.go:122] "RemoveContainer" containerID="7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76" Nov 03 09:15:16.416323 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:16.416301 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76\": container with ID starting with 7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76 not found: ID does not exist" containerID="7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76" Nov 03 09:15:16.416400 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.416326 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76"} err="failed to get container status \"7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76\": rpc error: code = NotFound desc = could not find container \"7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76\": container with ID starting with 7a8c58cc5eaff895a91ff88ce940c8e8fce43c3f9fbac96739c14e2e948ddd76 not found: ID does not exist" Nov 03 09:15:16.439953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.439923 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e9f4e5-1c70-40c9-a56d-dff029b79b11-kube-api-access-kq9pw" pod "28e9f4e5-1c70-40c9-a56d-dff029b79b11" (UID: "28e9f4e5-1c70-40c9-a56d-dff029b79b11"). InnerVolumeSpecName "kube-api-access-kq9pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:16.508629 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.508591 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq9pw\" (UniqueName: \"kubernetes.io/projected/28e9f4e5-1c70-40c9-a56d-dff029b79b11-kube-api-access-kq9pw\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:16.534699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.534661 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" podUID="0ad07039-b190-421f-8c5f-32be14742a24" containerName="payload-pre-processing" containerID="cri-o://5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a" gracePeriod=25 Nov 03 09:15:16.823353 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.823212 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:15:16.828282 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.828253 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx"] Nov 03 09:15:16.832105 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.832083 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-7fc7b54847-mhmjx"] Nov 03 09:15:16.905666 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:16.905631 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:15:16.914126 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.914101 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad07039-b190-421f-8c5f-32be14742a24-kube-api-access-c9pgz\" (UniqueName: \"kubernetes.io/projected/0ad07039-b190-421f-8c5f-32be14742a24-kube-api-access-c9pgz\") pod \"0ad07039-b190-421f-8c5f-32be14742a24\" (UID: \"0ad07039-b190-421f-8c5f-32be14742a24\") " Nov 03 09:15:16.914363 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.914347 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad07039-b190-421f-8c5f-32be14742a24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad07039-b190-421f-8c5f-32be14742a24-plugins-config-volume\") pod \"0ad07039-b190-421f-8c5f-32be14742a24\" (UID: \"0ad07039-b190-421f-8c5f-32be14742a24\") " Nov 03 09:15:16.914737 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.914706 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad07039-b190-421f-8c5f-32be14742a24-plugins-config-volume" pod "0ad07039-b190-421f-8c5f-32be14742a24" (UID: "0ad07039-b190-421f-8c5f-32be14742a24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:16.914910 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.914895 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad07039-b190-421f-8c5f-32be14742a24-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:16.916338 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:16.916310 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad07039-b190-421f-8c5f-32be14742a24-kube-api-access-c9pgz" pod "0ad07039-b190-421f-8c5f-32be14742a24" (UID: "0ad07039-b190-421f-8c5f-32be14742a24"). InnerVolumeSpecName "kube-api-access-c9pgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:17.016212 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.016177 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9pgz\" (UniqueName: \"kubernetes.io/projected/0ad07039-b190-421f-8c5f-32be14742a24-kube-api-access-c9pgz\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:17.203691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.203615 5015 scope.go:122] "RemoveContainer" containerID="8ac0f68a6c5676e96e3b87b0db70de17bf62afd79effdfa129afe4dda056029e" Nov 03 09:15:17.208694 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.208664 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e9f4e5-1c70-40c9-a56d-dff029b79b11" path="/var/lib/kubelet/pods/28e9f4e5-1c70-40c9-a56d-dff029b79b11/volumes" Nov 03 09:15:17.410710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.410674 5015 generic.go:356] "Generic (PLEG): container finished" podID="0ad07039-b190-421f-8c5f-32be14742a24" containerID="5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a" exitCode=0 Nov 03 09:15:17.410879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.410730 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" Nov 03 09:15:17.410879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.410753 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" event={"ID":"0ad07039-b190-421f-8c5f-32be14742a24","Type":"ContainerDied","Data":"5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a"} Nov 03 09:15:17.410879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.410837 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7" event={"ID":"0ad07039-b190-421f-8c5f-32be14742a24","Type":"ContainerDied","Data":"ae961c1d1213617434409ed8ed704500fa14d9fa79d0ed99f902c6f575a1b6bc"} Nov 03 09:15:17.410879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.410853 5015 scope.go:122] "RemoveContainer" containerID="5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a" Nov 03 09:15:17.412980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.412950 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" event={"ID":"8c093ebc-02f0-4353-84f8-4aca0ceadaa6","Type":"ContainerStarted","Data":"15f6d12537c00afa99a0e9c77029531d986fbecd64154506a7b9984f2991b454"} Nov 03 09:15:17.416071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.416051 5015 generic.go:356] "Generic (PLEG): container finished" podID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerID="46179f4f3a5cd4c61fabfb9dc2357571b452897c53a3d6444362b44cbe5df137" exitCode=60 Nov 03 09:15:17.416158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.416119 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" event={"ID":"455cdb19-5da4-47fd-96be-49c63d856fe8","Type":"ContainerDied","Data":"46179f4f3a5cd4c61fabfb9dc2357571b452897c53a3d6444362b44cbe5df137"} Nov 03 09:15:17.416470 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.416453 5015 scope.go:122] "RemoveContainer" containerID="46179f4f3a5cd4c61fabfb9dc2357571b452897c53a3d6444362b44cbe5df137" Nov 03 09:15:17.416826 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:17.416796 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc_odh-ai-gateway-infra(455cdb19-5da4-47fd-96be-49c63d856fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" Nov 03 09:15:17.417955 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.417935 5015 generic.go:356] "Generic (PLEG): container finished" podID="ce533940-7370-468a-b2b9-48b34440f75c" containerID="fb04e202940029505f67b42bc3d8c02035bedd09ba138de0fd66df63657b4a0b" exitCode=60 Nov 03 09:15:17.418039 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.418008 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" event={"ID":"ce533940-7370-468a-b2b9-48b34440f75c","Type":"ContainerDied","Data":"fb04e202940029505f67b42bc3d8c02035bedd09ba138de0fd66df63657b4a0b"} Nov 03 09:15:17.418317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.418304 5015 scope.go:122] "RemoveContainer" containerID="fb04e202940029505f67b42bc3d8c02035bedd09ba138de0fd66df63657b4a0b" Nov 03 09:15:17.418564 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:17.418547 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv_odh-ai-gateway-infra(ce533940-7370-468a-b2b9-48b34440f75c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" podUID="ce533940-7370-468a-b2b9-48b34440f75c" Nov 03 09:15:17.492688 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.492652 5015 scope.go:122] "RemoveContainer" containerID="5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a" Nov 03 09:15:17.493180 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:17.493149 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a\": container with ID starting with 5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a not found: ID does not exist" containerID="5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a" Nov 03 09:15:17.493298 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.493196 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a"} err="failed to get container status \"5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a\": rpc error: code = NotFound desc = could not find container \"5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a\": container with ID starting with 5a7ca84fd022edbd5c050e5ed019fc819032a488dfdfc23ff3c120d6863c845a not found: ID does not exist" Nov 03 09:15:17.493298 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.493223 5015 scope.go:122] "RemoveContainer" containerID="d3ce0934e793aef2e5dee472a3397c007a930a9e8300efe9b211d614a3d8e3a6" Nov 03 09:15:17.545235 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.545116 5015 scope.go:122] "RemoveContainer" containerID="8bd0ffcc366baeebd072630173af504f0a6a2b53e341b3f6ae445051574abf3b" Nov 03 09:15:17.934194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.934100 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7"] Nov 03 09:15:17.938363 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:17.938329 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-6b4fb4455f-ksvf7"] Nov 03 09:15:18.429779 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.429748 5015 generic.go:356] "Generic (PLEG): container finished" podID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerID="15f6d12537c00afa99a0e9c77029531d986fbecd64154506a7b9984f2991b454" exitCode=60 Nov 03 09:15:18.430277 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.429825 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" event={"ID":"8c093ebc-02f0-4353-84f8-4aca0ceadaa6","Type":"ContainerDied","Data":"15f6d12537c00afa99a0e9c77029531d986fbecd64154506a7b9984f2991b454"} Nov 03 09:15:18.430277 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.429887 5015 scope.go:122] "RemoveContainer" containerID="d0c7901caa1bd2423701d17dcb4a30c5a6f2ad57c7f56768043851e7e7f8f155" Nov 03 09:15:18.430373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.430280 5015 scope.go:122] "RemoveContainer" containerID="15f6d12537c00afa99a0e9c77029531d986fbecd64154506a7b9984f2991b454" Nov 03 09:15:18.430685 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:18.430652 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766_odh-ai-gateway-infra(8c093ebc-02f0-4353-84f8-4aca0ceadaa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" Nov 03 09:15:18.432061 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.432038 5015 generic.go:356] "Generic (PLEG): container finished" podID="184e292a-8c9f-42f1-be46-57313bd4e311" containerID="3d4bc48c3fb37b4fbbd8ad78e2ad632e2e1506a2bd9f73e448fba53cd4bf2a44" exitCode=60 Nov 03 09:15:18.432161 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.432106 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" event={"ID":"184e292a-8c9f-42f1-be46-57313bd4e311","Type":"ContainerDied","Data":"3d4bc48c3fb37b4fbbd8ad78e2ad632e2e1506a2bd9f73e448fba53cd4bf2a44"} Nov 03 09:15:18.432555 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.432534 5015 scope.go:122] "RemoveContainer" containerID="3d4bc48c3fb37b4fbbd8ad78e2ad632e2e1506a2bd9f73e448fba53cd4bf2a44" Nov 03 09:15:18.432903 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:18.432882 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894955-v9xxn_odh-ai-gateway-infra(184e292a-8c9f-42f1-be46-57313bd4e311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" Nov 03 09:15:18.463603 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.463572 5015 scope.go:122] "RemoveContainer" containerID="8ac0f68a6c5676e96e3b87b0db70de17bf62afd79effdfa129afe4dda056029e" Nov 03 09:15:18.607258 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.607169 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766"] Nov 03 09:15:18.718984 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.718954 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc"] Nov 03 09:15:18.834115 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.834079 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv"] Nov 03 09:15:18.916319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:18.914343 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" Nov 03 09:15:19.017238 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.017213 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" Nov 03 09:15:19.068094 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.068066 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/455cdb19-5da4-47fd-96be-49c63d856fe8-kube-api-access-g44fk\" (UniqueName: \"kubernetes.io/projected/455cdb19-5da4-47fd-96be-49c63d856fe8-kube-api-access-g44fk\") pod \"455cdb19-5da4-47fd-96be-49c63d856fe8\" (UID: \"455cdb19-5da4-47fd-96be-49c63d856fe8\") " Nov 03 09:15:19.068291 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.068165 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/455cdb19-5da4-47fd-96be-49c63d856fe8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455cdb19-5da4-47fd-96be-49c63d856fe8-maas-api-service-ca\") pod \"455cdb19-5da4-47fd-96be-49c63d856fe8\" (UID: \"455cdb19-5da4-47fd-96be-49c63d856fe8\") " Nov 03 09:15:19.068644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.068614 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455cdb19-5da4-47fd-96be-49c63d856fe8-maas-api-service-ca" pod "455cdb19-5da4-47fd-96be-49c63d856fe8" (UID: "455cdb19-5da4-47fd-96be-49c63d856fe8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:19.068797 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.068724 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455cdb19-5da4-47fd-96be-49c63d856fe8-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:19.070631 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.070606 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455cdb19-5da4-47fd-96be-49c63d856fe8-kube-api-access-g44fk" pod "455cdb19-5da4-47fd-96be-49c63d856fe8" (UID: "455cdb19-5da4-47fd-96be-49c63d856fe8"). InnerVolumeSpecName "kube-api-access-g44fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:19.169601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.169507 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce533940-7370-468a-b2b9-48b34440f75c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce533940-7370-468a-b2b9-48b34440f75c-maas-api-service-ca\") pod \"ce533940-7370-468a-b2b9-48b34440f75c\" (UID: \"ce533940-7370-468a-b2b9-48b34440f75c\") " Nov 03 09:15:19.169766 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.169616 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce533940-7370-468a-b2b9-48b34440f75c-kube-api-access-zrpgb\" (UniqueName: \"kubernetes.io/projected/ce533940-7370-468a-b2b9-48b34440f75c-kube-api-access-zrpgb\") pod \"ce533940-7370-468a-b2b9-48b34440f75c\" (UID: \"ce533940-7370-468a-b2b9-48b34440f75c\") " Nov 03 09:15:19.169917 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.169891 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce533940-7370-468a-b2b9-48b34440f75c-maas-api-service-ca" pod "ce533940-7370-468a-b2b9-48b34440f75c" (UID: "ce533940-7370-468a-b2b9-48b34440f75c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:19.170107 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.170091 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g44fk\" (UniqueName: \"kubernetes.io/projected/455cdb19-5da4-47fd-96be-49c63d856fe8-kube-api-access-g44fk\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:19.170160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.170117 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce533940-7370-468a-b2b9-48b34440f75c-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:19.171969 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.171943 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce533940-7370-468a-b2b9-48b34440f75c-kube-api-access-zrpgb" pod "ce533940-7370-468a-b2b9-48b34440f75c" (UID: "ce533940-7370-468a-b2b9-48b34440f75c"). InnerVolumeSpecName "kube-api-access-zrpgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:19.210840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.210803 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad07039-b190-421f-8c5f-32be14742a24" path="/var/lib/kubelet/pods/0ad07039-b190-421f-8c5f-32be14742a24/volumes" Nov 03 09:15:19.271451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.271419 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrpgb\" (UniqueName: \"kubernetes.io/projected/ce533940-7370-468a-b2b9-48b34440f75c-kube-api-access-zrpgb\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:19.295063 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:19.295025 5015 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455cdb19_5da4_47fd_96be_49c63d856fe8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce533940_7370_468a_b2b9_48b34440f75c.slice\": RecentStats: unable to find data in memory cache]" Nov 03 09:15:19.446520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.446117 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" event={"ID":"ce533940-7370-468a-b2b9-48b34440f75c","Type":"ContainerDied","Data":"b574e897287c6f831475323c5debc56b6129cbf09b09d64611561b159e6db06b"} Nov 03 09:15:19.446520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.446269 5015 scope.go:122] "RemoveContainer" containerID="fb04e202940029505f67b42bc3d8c02035bedd09ba138de0fd66df63657b4a0b" Nov 03 09:15:19.447032 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.446591 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv" Nov 03 09:15:19.455829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.455801 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" event={"ID":"455cdb19-5da4-47fd-96be-49c63d856fe8","Type":"ContainerDied","Data":"375710ab405b4ad012f064b29f097e30088b8538346e4406d022a3d2093532da"} Nov 03 09:15:19.455960 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.455943 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc" Nov 03 09:15:19.474436 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.467220 5015 scope.go:122] "RemoveContainer" containerID="46179f4f3a5cd4c61fabfb9dc2357571b452897c53a3d6444362b44cbe5df137" Nov 03 09:15:19.630030 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.630002 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" Nov 03 09:15:19.696239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.696211 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn"] Nov 03 09:15:19.732336 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.732304 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc"] Nov 03 09:15:19.743603 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.743560 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-77da2a-29894955-lmqsc"] Nov 03 09:15:19.786868 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.786038 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-kube-api-access-spz9f\" (UniqueName: \"kubernetes.io/projected/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-kube-api-access-spz9f\") pod \"8c093ebc-02f0-4353-84f8-4aca0ceadaa6\" (UID: \"8c093ebc-02f0-4353-84f8-4aca0ceadaa6\") " Nov 03 09:15:19.786868 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.786186 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-maas-api-service-ca\") pod \"8c093ebc-02f0-4353-84f8-4aca0ceadaa6\" (UID: \"8c093ebc-02f0-4353-84f8-4aca0ceadaa6\") " Nov 03 09:15:19.786868 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.786540 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-maas-api-service-ca" pod "8c093ebc-02f0-4353-84f8-4aca0ceadaa6" (UID: "8c093ebc-02f0-4353-84f8-4aca0ceadaa6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:19.786868 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.786783 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:19.789685 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.789657 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-kube-api-access-spz9f" pod "8c093ebc-02f0-4353-84f8-4aca0ceadaa6" (UID: "8c093ebc-02f0-4353-84f8-4aca0ceadaa6"). InnerVolumeSpecName "kube-api-access-spz9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:19.856537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.856501 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv"] Nov 03 09:15:19.863008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.862976 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c32d41-29894955-8rclv"] Nov 03 09:15:19.868080 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.868056 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" Nov 03 09:15:19.887747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.887717 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spz9f\" (UniqueName: \"kubernetes.io/projected/8c093ebc-02f0-4353-84f8-4aca0ceadaa6-kube-api-access-spz9f\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:19.988302 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.988272 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184e292a-8c9f-42f1-be46-57313bd4e311-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/184e292a-8c9f-42f1-be46-57313bd4e311-maas-api-service-ca\") pod \"184e292a-8c9f-42f1-be46-57313bd4e311\" (UID: \"184e292a-8c9f-42f1-be46-57313bd4e311\") " Nov 03 09:15:19.988510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.988377 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184e292a-8c9f-42f1-be46-57313bd4e311-kube-api-access-4q6fl\" (UniqueName: \"kubernetes.io/projected/184e292a-8c9f-42f1-be46-57313bd4e311-kube-api-access-4q6fl\") pod \"184e292a-8c9f-42f1-be46-57313bd4e311\" (UID: \"184e292a-8c9f-42f1-be46-57313bd4e311\") " Nov 03 09:15:19.988749 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.988722 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184e292a-8c9f-42f1-be46-57313bd4e311-maas-api-service-ca" pod "184e292a-8c9f-42f1-be46-57313bd4e311" (UID: "184e292a-8c9f-42f1-be46-57313bd4e311"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:19.988892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.988878 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/184e292a-8c9f-42f1-be46-57313bd4e311-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:19.990717 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:19.990697 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184e292a-8c9f-42f1-be46-57313bd4e311-kube-api-access-4q6fl" pod "184e292a-8c9f-42f1-be46-57313bd4e311" (UID: "184e292a-8c9f-42f1-be46-57313bd4e311"). InnerVolumeSpecName "kube-api-access-4q6fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:20.090694 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:20.090661 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q6fl\" (UniqueName: \"kubernetes.io/projected/184e292a-8c9f-42f1-be46-57313bd4e311-kube-api-access-4q6fl\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:20.469205 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:20.469157 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" Nov 03 09:15:20.469763 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:20.469198 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766" event={"ID":"8c093ebc-02f0-4353-84f8-4aca0ceadaa6","Type":"ContainerDied","Data":"df4984f055f596abc2d4b657c7b4d95f68800e43b5e3e18820e8ebde4efe60f8"} Nov 03 09:15:20.469763 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:20.469274 5015 scope.go:122] "RemoveContainer" containerID="15f6d12537c00afa99a0e9c77029531d986fbecd64154506a7b9984f2991b454" Nov 03 09:15:20.471384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:20.471367 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" Nov 03 09:15:20.471641 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:20.471408 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn" event={"ID":"184e292a-8c9f-42f1-be46-57313bd4e311","Type":"ContainerDied","Data":"8a580997524fcb39b37045d39bcfdf6b8899080b03f1c38edf5b4f715160a11b"} Nov 03 09:15:20.483194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:20.483178 5015 scope.go:122] "RemoveContainer" containerID="3d4bc48c3fb37b4fbbd8ad78e2ad632e2e1506a2bd9f73e448fba53cd4bf2a44" Nov 03 09:15:20.643033 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:20.642945 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn"] Nov 03 09:15:20.651277 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:20.651244 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894955-v9xxn"] Nov 03 09:15:20.763729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:20.763692 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766"] Nov 03 09:15:20.767088 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:20.767065 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894955-zt766"] Nov 03 09:15:21.205618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:21.205587 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" path="/var/lib/kubelet/pods/184e292a-8c9f-42f1-be46-57313bd4e311/volumes" Nov 03 09:15:21.206033 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:21.206019 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" path="/var/lib/kubelet/pods/455cdb19-5da4-47fd-96be-49c63d856fe8/volumes" Nov 03 09:15:21.206425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:21.206412 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" path="/var/lib/kubelet/pods/8c093ebc-02f0-4353-84f8-4aca0ceadaa6/volumes" Nov 03 09:15:21.207117 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:21.207102 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce533940-7370-468a-b2b9-48b34440f75c" path="/var/lib/kubelet/pods/ce533940-7370-468a-b2b9-48b34440f75c/volumes" Nov 03 09:15:45.288467 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.288421 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9"] Nov 03 09:15:45.289010 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.288728 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" podUID="ea7468ac-f9fb-43ba-b333-6644840e9919" containerName="payload-processing" containerID="cri-o://d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e" gracePeriod=30 Nov 03 09:15:45.344040 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.344013 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5"] Nov 03 09:15:45.344228 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.344206 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" podUID="615d46d0-2300-496e-aa08-9982e5dc99b2" containerName="payload-pre-processing" containerID="cri-o://3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca" gracePeriod=30 Nov 03 09:15:45.624683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.624654 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:15:45.658953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.658929 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:15:45.687659 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.687629 5015 generic.go:356] "Generic (PLEG): container finished" podID="ea7468ac-f9fb-43ba-b333-6644840e9919" containerID="d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e" exitCode=0 Nov 03 09:15:45.687803 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.687670 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" Nov 03 09:15:45.687803 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.687716 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" event={"ID":"ea7468ac-f9fb-43ba-b333-6644840e9919","Type":"ContainerDied","Data":"d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e"} Nov 03 09:15:45.687803 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.687769 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9" event={"ID":"ea7468ac-f9fb-43ba-b333-6644840e9919","Type":"ContainerDied","Data":"4280beb809341b3f423344cb17721ec7c25b9d3bca48360d0f997bdc22856746"} Nov 03 09:15:45.687803 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.687798 5015 scope.go:122] "RemoveContainer" containerID="d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e" Nov 03 09:15:45.689332 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.689307 5015 generic.go:356] "Generic (PLEG): container finished" podID="615d46d0-2300-496e-aa08-9982e5dc99b2" containerID="3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca" exitCode=0 Nov 03 09:15:45.689460 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.689364 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" Nov 03 09:15:45.689460 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.689421 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" event={"ID":"615d46d0-2300-496e-aa08-9982e5dc99b2","Type":"ContainerDied","Data":"3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca"} Nov 03 09:15:45.689460 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.689459 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5" event={"ID":"615d46d0-2300-496e-aa08-9982e5dc99b2","Type":"ContainerDied","Data":"0d55def18f7278b529611febd1dcc5b12c290cfd11047ec6ef2a1eb98bec85b9"} Nov 03 09:15:45.699026 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.699001 5015 scope.go:122] "RemoveContainer" containerID="d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e" Nov 03 09:15:45.699346 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:45.699323 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e\": container with ID starting with d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e not found: ID does not exist" containerID="d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e" Nov 03 09:15:45.699447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.699355 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e"} err="failed to get container status \"d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e\": rpc error: code = NotFound desc = could not find container \"d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e\": container with ID starting with d8d0e17526c919caf5a5b8d3625e8083c8d25412897fe504e0df6e4199cc639e not found: ID does not exist" Nov 03 09:15:45.699447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.699373 5015 scope.go:122] "RemoveContainer" containerID="3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca" Nov 03 09:15:45.710721 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.710697 5015 scope.go:122] "RemoveContainer" containerID="3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca" Nov 03 09:15:45.711065 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:45.711033 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca\": container with ID starting with 3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca not found: ID does not exist" containerID="3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca" Nov 03 09:15:45.711138 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.711069 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca"} err="failed to get container status \"3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca\": rpc error: code = NotFound desc = could not find container \"3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca\": container with ID starting with 3b9e0deb8d3b6bde27d72a6a605e045c933f8b569136d610194be74277365cca not found: ID does not exist" Nov 03 09:15:45.755359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.754622 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615d46d0-2300-496e-aa08-9982e5dc99b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615d46d0-2300-496e-aa08-9982e5dc99b2-plugins-config-volume\") pod \"615d46d0-2300-496e-aa08-9982e5dc99b2\" (UID: \"615d46d0-2300-496e-aa08-9982e5dc99b2\") " Nov 03 09:15:45.755359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.754718 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7468ac-f9fb-43ba-b333-6644840e9919-kube-api-access-xqf5d\" (UniqueName: \"kubernetes.io/projected/ea7468ac-f9fb-43ba-b333-6644840e9919-kube-api-access-xqf5d\") pod \"ea7468ac-f9fb-43ba-b333-6644840e9919\" (UID: \"ea7468ac-f9fb-43ba-b333-6644840e9919\") " Nov 03 09:15:45.755359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.754751 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615d46d0-2300-496e-aa08-9982e5dc99b2-kube-api-access-2zppz\" (UniqueName: \"kubernetes.io/projected/615d46d0-2300-496e-aa08-9982e5dc99b2-kube-api-access-2zppz\") pod \"615d46d0-2300-496e-aa08-9982e5dc99b2\" (UID: \"615d46d0-2300-496e-aa08-9982e5dc99b2\") " Nov 03 09:15:45.755359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.754987 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7468ac-f9fb-43ba-b333-6644840e9919-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7468ac-f9fb-43ba-b333-6644840e9919-plugins-config-volume\") pod \"ea7468ac-f9fb-43ba-b333-6644840e9919\" (UID: \"ea7468ac-f9fb-43ba-b333-6644840e9919\") " Nov 03 09:15:45.755695 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.754986 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615d46d0-2300-496e-aa08-9982e5dc99b2-plugins-config-volume" pod "615d46d0-2300-496e-aa08-9982e5dc99b2" (UID: "615d46d0-2300-496e-aa08-9982e5dc99b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:45.755695 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.755456 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7468ac-f9fb-43ba-b333-6644840e9919-plugins-config-volume" pod "ea7468ac-f9fb-43ba-b333-6644840e9919" (UID: "ea7468ac-f9fb-43ba-b333-6644840e9919"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:45.756292 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.756268 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7468ac-f9fb-43ba-b333-6644840e9919-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:45.756419 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.756403 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615d46d0-2300-496e-aa08-9982e5dc99b2-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:45.759603 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.757973 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615d46d0-2300-496e-aa08-9982e5dc99b2-kube-api-access-2zppz" pod "615d46d0-2300-496e-aa08-9982e5dc99b2" (UID: "615d46d0-2300-496e-aa08-9982e5dc99b2"). InnerVolumeSpecName "kube-api-access-2zppz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:45.759603 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.759251 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7468ac-f9fb-43ba-b333-6644840e9919-kube-api-access-xqf5d" pod "ea7468ac-f9fb-43ba-b333-6644840e9919" (UID: "ea7468ac-f9fb-43ba-b333-6644840e9919"). InnerVolumeSpecName "kube-api-access-xqf5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:45.874560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.873572 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqf5d\" (UniqueName: \"kubernetes.io/projected/ea7468ac-f9fb-43ba-b333-6644840e9919-kube-api-access-xqf5d\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:45.874560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:45.873612 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zppz\" (UniqueName: \"kubernetes.io/projected/615d46d0-2300-496e-aa08-9982e5dc99b2-kube-api-access-2zppz\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:46.118347 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:46.118314 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5"] Nov 03 09:15:46.122933 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:46.122906 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77da2a-7d55df5bd7-dw7w5"] Nov 03 09:15:46.236363 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:46.236327 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9"] Nov 03 09:15:46.239894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:46.239871 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77da2a-64cccd5df8-scrk9"] Nov 03 09:15:46.904056 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:15:46.904012 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:15:47.205377 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.205296 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615d46d0-2300-496e-aa08-9982e5dc99b2" path="/var/lib/kubelet/pods/615d46d0-2300-496e-aa08-9982e5dc99b2/volumes" Nov 03 09:15:47.205791 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.205772 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7468ac-f9fb-43ba-b333-6644840e9919" path="/var/lib/kubelet/pods/ea7468ac-f9fb-43ba-b333-6644840e9919/volumes" Nov 03 09:15:47.688413 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.688381 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7"] Nov 03 09:15:47.689042 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689025 5015 cpu_manager.go:414] "containerMap: removing container" podUID="f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" containerName="cleanup" Nov 03 09:15:47.689098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689045 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" containerName="cleanup" Nov 03 09:15:47.689098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689059 5015 cpu_manager.go:414] "containerMap: removing container" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" containerName="cleanup" Nov 03 09:15:47.689098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689063 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" containerName="cleanup" Nov 03 09:15:47.689098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689072 5015 cpu_manager.go:414] "containerMap: removing container" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerName="cleanup" Nov 03 09:15:47.689098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689077 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerName="cleanup" Nov 03 09:15:47.689098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689085 5015 cpu_manager.go:414] "containerMap: removing container" podUID="ce533940-7370-468a-b2b9-48b34440f75c" containerName="cleanup" Nov 03 09:15:47.689098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689089 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce533940-7370-468a-b2b9-48b34440f75c" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689108 5015 cpu_manager.go:414] "containerMap: removing container" podUID="28e9f4e5-1c70-40c9-a56d-dff029b79b11" containerName="payload-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689112 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e9f4e5-1c70-40c9-a56d-dff029b79b11" containerName="payload-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689120 5015 cpu_manager.go:414] "containerMap: removing container" podUID="f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689125 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689129 5015 cpu_manager.go:414] "containerMap: removing container" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689133 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689142 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689146 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689156 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689160 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689168 5015 cpu_manager.go:414] "containerMap: removing container" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689172 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689180 5015 cpu_manager.go:414] "containerMap: removing container" podUID="ea7468ac-f9fb-43ba-b333-6644840e9919" containerName="payload-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689185 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7468ac-f9fb-43ba-b333-6644840e9919" containerName="payload-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689191 5015 cpu_manager.go:414] "containerMap: removing container" podUID="615d46d0-2300-496e-aa08-9982e5dc99b2" containerName="payload-pre-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689195 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615d46d0-2300-496e-aa08-9982e5dc99b2" containerName="payload-pre-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689202 5015 cpu_manager.go:414] "containerMap: removing container" podUID="569b803e-986a-47b3-8552-3f80dd115e23" containerName="payload-pre-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689206 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569b803e-986a-47b3-8552-3f80dd115e23" containerName="payload-pre-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689212 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689217 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689227 5015 cpu_manager.go:414] "containerMap: removing container" podUID="ce533940-7370-468a-b2b9-48b34440f75c" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689231 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce533940-7370-468a-b2b9-48b34440f75c" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689238 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0ad07039-b190-421f-8c5f-32be14742a24" containerName="payload-pre-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689243 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad07039-b190-421f-8c5f-32be14742a24" containerName="payload-pre-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689254 5015 cpu_manager.go:414] "containerMap: removing container" podUID="24a0503c-ea10-46b4-91cc-665171c35c2c" containerName="payload-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689258 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a0503c-ea10-46b4-91cc-665171c35c2c" containerName="payload-processing" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689264 5015 cpu_manager.go:414] "containerMap: removing container" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerName="cleanup" Nov 03 09:15:47.689306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689268 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerName="cleanup" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689383 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e9f4e5-1c70-40c9-a56d-dff029b79b11" containerName="payload-processing" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689395 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" containerName="cleanup" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689401 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" containerName="cleanup" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689408 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerName="cleanup" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689415 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerName="cleanup" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689421 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerName="cleanup" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689427 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" containerName="cleanup" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689432 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7468ac-f9fb-43ba-b333-6644840e9919" containerName="payload-processing" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689437 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce533940-7370-468a-b2b9-48b34440f75c" containerName="cleanup" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689442 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="615d46d0-2300-496e-aa08-9982e5dc99b2" containerName="payload-pre-processing" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689448 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="569b803e-986a-47b3-8552-3f80dd115e23" containerName="payload-pre-processing" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689453 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c093ebc-02f0-4353-84f8-4aca0ceadaa6" containerName="cleanup" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689459 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce533940-7370-468a-b2b9-48b34440f75c" containerName="cleanup" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689464 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad07039-b190-421f-8c5f-32be14742a24" containerName="payload-pre-processing" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689470 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a0503c-ea10-46b4-91cc-665171c35c2c" containerName="payload-processing" Nov 03 09:15:47.690275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.689491 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerName="cleanup" Nov 03 09:15:47.694594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.694575 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.785336 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.785303 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7"] Nov 03 09:15:47.799261 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.799211 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.799446 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.799303 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf17300-8bf7-445d-b29d-30774666a981-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.799446 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.799363 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.799446 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.799387 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.799446 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.799430 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.799722 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.799457 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8w6f\" (UniqueName: \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-kube-api-access-x8w6f\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.799722 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.799541 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf17300-8bf7-445d-b29d-30774666a981-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.799722 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.799569 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.799722 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.799671 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.900934 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.900897 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901134 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901030 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901134 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901112 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf17300-8bf7-445d-b29d-30774666a981-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901160 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901188 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901231 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901415 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901259 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8w6f\" (UniqueName: \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-kube-api-access-x8w6f\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901415 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901286 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901415 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901300 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf17300-8bf7-445d-b29d-30774666a981-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901415 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901326 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901741 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901469 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901741 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901612 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.901741 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.901731 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.902067 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.902034 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf17300-8bf7-445d-b29d-30774666a981-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.903640 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.903616 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.903843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.903823 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf17300-8bf7-445d-b29d-30774666a981-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.961528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.961396 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:47.961998 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:47.961750 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8w6f\" (UniqueName: \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-kube-api-access-x8w6f\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:48.008971 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:48.008942 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:48.454196 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:15:48.454155 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf17300_8bf7_445d_b29d_30774666a981.slice/crio-e5f37ce62553d5b6cbd6e4d2dbff1276f7337c42e9b791626e58c9f6c8810d32 WatchSource:0}: Error finding container e5f37ce62553d5b6cbd6e4d2dbff1276f7337c42e9b791626e58c9f6c8810d32: Status 404 returned error can't find the container with id e5f37ce62553d5b6cbd6e4d2dbff1276f7337c42e9b791626e58c9f6c8810d32 Nov 03 09:15:48.455272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:48.455242 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7"] Nov 03 09:15:48.457518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:48.457445 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:15:48.457609 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:48.457565 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:15:48.457646 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:48.457609 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:15:48.717818 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:48.717720 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" event={"ID":"dcf17300-8bf7-445d-b29d-30774666a981","Type":"ContainerStarted","Data":"8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465"} Nov 03 09:15:48.717818 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:48.717784 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" event={"ID":"dcf17300-8bf7-445d-b29d-30774666a981","Type":"ContainerStarted","Data":"e5f37ce62553d5b6cbd6e4d2dbff1276f7337c42e9b791626e58c9f6c8810d32"} Nov 03 09:15:48.848320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:48.848252 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" podStartSLOduration=1.848229914 podStartE2EDuration="1.848229914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:48.842765647 +0000 UTC m=+2181.767707864" watchObservedRunningTime="2026-11-03 09:15:48.848229914 +0000 UTC m=+2181.773172132" Nov 03 09:15:49.009778 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:49.009747 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:49.011309 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:49.011282 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 09:15:49.011453 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:49.011330 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 09:15:50.009857 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:50.009819 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 09:15:50.010348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:50.009895 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 09:15:51.010167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:51.010126 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 09:15:51.010626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:51.010199 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 09:15:52.009288 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:52.009251 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 09:15:52.009554 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:52.009310 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 09:15:53.009424 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:53.009384 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 09:15:53.009982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:53.009467 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 09:15:54.011602 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:54.011549 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 09:15:54.012341 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:54.012289 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 09:15:55.010420 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:55.010363 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 09:15:55.010665 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:55.010435 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 09:15:56.024878 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:56.024086 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 09:15:56.024878 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:56.024154 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 09:15:58.010361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:58.010266 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:15:58.010361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:58.010324 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:15:58.067657 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:58.067620 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:58.068390 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:58.068365 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:15:58.069465 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:15:58.069437 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:16:08.497032 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.496952 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk"] Nov 03 09:16:08.497620 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.497605 5015 cpu_manager.go:414] "containerMap: removing container" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerName="cleanup" Nov 03 09:16:08.497684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.497621 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerName="cleanup" Nov 03 09:16:08.497684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.497630 5015 cpu_manager.go:414] "containerMap: removing container" podUID="ce533940-7370-468a-b2b9-48b34440f75c" containerName="cleanup" Nov 03 09:16:08.497684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.497637 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce533940-7370-468a-b2b9-48b34440f75c" containerName="cleanup" Nov 03 09:16:08.497786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.497733 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ac62f3-e0bb-4ea9-9b12-fe81de0600e5" containerName="cleanup" Nov 03 09:16:08.497786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.497741 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="184e292a-8c9f-42f1-be46-57313bd4e311" containerName="cleanup" Nov 03 09:16:08.497786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.497749 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="455cdb19-5da4-47fd-96be-49c63d856fe8" containerName="cleanup" Nov 03 09:16:08.497786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.497760 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce533940-7370-468a-b2b9-48b34440f75c" containerName="cleanup" Nov 03 09:16:08.505786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.505754 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.531327 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.531298 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.531493 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.531403 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.531493 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.531429 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.531621 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.531566 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.531621 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.531604 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xgkq\" (UniqueName: \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-kube-api-access-2xgkq\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.531781 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.531759 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.531823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.531806 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.531892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.531833 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.531932 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.531895 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.616033 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.615995 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk"] Nov 03 09:16:08.633574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.633537 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xgkq\" (UniqueName: \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-kube-api-access-2xgkq\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.633759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.633660 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.633759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.633707 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.633759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.633740 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.633952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.633771 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.633952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.633837 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.633952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.633912 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.633952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.633937 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.634170 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.633993 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.634170 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.634138 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.634278 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.634227 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.635303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.635277 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.635433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.635330 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.635822 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.635797 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.637823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.637801 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.638101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.638077 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.698982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.698944 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.699139 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.699051 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xgkq\" (UniqueName: \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-kube-api-access-2xgkq\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:08.823594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:08.823504 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:09.256672 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:09.256642 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk"] Nov 03 09:16:09.257751 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:16:09.257725 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e72455e_e27f_41c7_99e0_6dcabd1a2882.slice/crio-2fe9e77912bf1ff9bbb5c051244fa3652e4faac5296bb969d6adedf966968433 WatchSource:0}: Error finding container 2fe9e77912bf1ff9bbb5c051244fa3652e4faac5296bb969d6adedf966968433: Status 404 returned error can't find the container with id 2fe9e77912bf1ff9bbb5c051244fa3652e4faac5296bb969d6adedf966968433 Nov 03 09:16:09.260317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:09.260284 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:09.260389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:09.260350 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:09.260389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:09.260378 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:09.977105 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:09.977068 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" event={"ID":"2e72455e-e27f-41c7-99e0-6dcabd1a2882","Type":"ContainerStarted","Data":"34e9c41a504757a171ffe186e72ebaa6a555725dcd3862b97ccb8c446640e506"} Nov 03 09:16:09.977659 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:09.977130 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" event={"ID":"2e72455e-e27f-41c7-99e0-6dcabd1a2882","Type":"ContainerStarted","Data":"2fe9e77912bf1ff9bbb5c051244fa3652e4faac5296bb969d6adedf966968433"} Nov 03 09:16:10.200561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:10.200511 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" podStartSLOduration=2.20049651 podStartE2EDuration="2.20049651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:10.150013479 +0000 UTC m=+2203.074955698" watchObservedRunningTime="2026-11-03 09:16:10.20049651 +0000 UTC m=+2203.125438723" Nov 03 09:16:10.824109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:10.824068 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:10.826066 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:10.826041 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 09:16:10.826215 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:10.826108 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 09:16:11.824088 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:11.824047 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 09:16:11.824498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:11.824110 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 09:16:12.824434 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:12.824368 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 09:16:12.824948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:12.824432 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 09:16:13.824374 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:13.824335 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 09:16:13.824585 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:13.824395 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 09:16:14.824093 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:14.824049 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 09:16:14.824269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:14.824113 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 09:16:15.824716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:15.824679 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 09:16:15.825105 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:15.824733 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 09:16:16.828149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:16.828120 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:16.828654 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:16.828598 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:16.829248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:16.829230 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:16:16.904074 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:16:16.904033 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:16:21.180935 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.180895 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68"] Nov 03 09:16:21.181392 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.181162 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" podUID="56530cb6-7d91-469a-96cb-4049f1ec5e5a" containerName="payload-processing" containerID="cri-o://44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec" gracePeriod=30 Nov 03 09:16:21.293717 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.293667 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs"] Nov 03 09:16:21.294230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.294160 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" podUID="a758c014-84c7-425e-b9cb-5ea42c09625d" containerName="payload-pre-processing" containerID="cri-o://229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd" gracePeriod=30 Nov 03 09:16:21.637995 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.636082 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:16:21.687650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.685250 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56530cb6-7d91-469a-96cb-4049f1ec5e5a-kube-api-access-62bcg\" (UniqueName: \"kubernetes.io/projected/56530cb6-7d91-469a-96cb-4049f1ec5e5a-kube-api-access-62bcg\") pod \"56530cb6-7d91-469a-96cb-4049f1ec5e5a\" (UID: \"56530cb6-7d91-469a-96cb-4049f1ec5e5a\") " Nov 03 09:16:21.687650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.685425 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56530cb6-7d91-469a-96cb-4049f1ec5e5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56530cb6-7d91-469a-96cb-4049f1ec5e5a-plugins-config-volume\") pod \"56530cb6-7d91-469a-96cb-4049f1ec5e5a\" (UID: \"56530cb6-7d91-469a-96cb-4049f1ec5e5a\") " Nov 03 09:16:21.687650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.686365 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56530cb6-7d91-469a-96cb-4049f1ec5e5a-plugins-config-volume" pod "56530cb6-7d91-469a-96cb-4049f1ec5e5a" (UID: "56530cb6-7d91-469a-96cb-4049f1ec5e5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:16:21.691604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.691571 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56530cb6-7d91-469a-96cb-4049f1ec5e5a-kube-api-access-62bcg" pod "56530cb6-7d91-469a-96cb-4049f1ec5e5a" (UID: "56530cb6-7d91-469a-96cb-4049f1ec5e5a"). InnerVolumeSpecName "kube-api-access-62bcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:21.801571 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.791878 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56530cb6-7d91-469a-96cb-4049f1ec5e5a-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:21.801571 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.791919 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62bcg\" (UniqueName: \"kubernetes.io/projected/56530cb6-7d91-469a-96cb-4049f1ec5e5a-kube-api-access-62bcg\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:21.965495 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:21.957056 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:16:22.057570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.057331 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a758c014-84c7-425e-b9cb-5ea42c09625d-kube-api-access-8lrrz\" (UniqueName: \"kubernetes.io/projected/a758c014-84c7-425e-b9cb-5ea42c09625d-kube-api-access-8lrrz\") pod \"a758c014-84c7-425e-b9cb-5ea42c09625d\" (UID: \"a758c014-84c7-425e-b9cb-5ea42c09625d\") " Nov 03 09:16:22.057570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.057422 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a758c014-84c7-425e-b9cb-5ea42c09625d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a758c014-84c7-425e-b9cb-5ea42c09625d-plugins-config-volume\") pod \"a758c014-84c7-425e-b9cb-5ea42c09625d\" (UID: \"a758c014-84c7-425e-b9cb-5ea42c09625d\") " Nov 03 09:16:22.117498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.116614 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a758c014-84c7-425e-b9cb-5ea42c09625d-plugins-config-volume" pod "a758c014-84c7-425e-b9cb-5ea42c09625d" (UID: "a758c014-84c7-425e-b9cb-5ea42c09625d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:16:22.287508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.236847 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a758c014-84c7-425e-b9cb-5ea42c09625d-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:22.336064 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.335978 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a758c014-84c7-425e-b9cb-5ea42c09625d-kube-api-access-8lrrz" pod "a758c014-84c7-425e-b9cb-5ea42c09625d" (UID: "a758c014-84c7-425e-b9cb-5ea42c09625d"). InnerVolumeSpecName "kube-api-access-8lrrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:22.338704 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.338673 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lrrz\" (UniqueName: \"kubernetes.io/projected/a758c014-84c7-425e-b9cb-5ea42c09625d-kube-api-access-8lrrz\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:22.485421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.485365 5015 generic.go:356] "Generic (PLEG): container finished" podID="a758c014-84c7-425e-b9cb-5ea42c09625d" containerID="229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd" exitCode=0 Nov 03 09:16:22.489013 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.488977 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" event={"ID":"a758c014-84c7-425e-b9cb-5ea42c09625d","Type":"ContainerDied","Data":"229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd"} Nov 03 09:16:22.489370 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.489351 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" event={"ID":"a758c014-84c7-425e-b9cb-5ea42c09625d","Type":"ContainerDied","Data":"1bdaefd6d9ef31707ec47d83557892c995f1994257da481f91639b67be68f6f3"} Nov 03 09:16:22.489574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.489555 5015 scope.go:122] "RemoveContainer" containerID="229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd" Nov 03 09:16:22.489788 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.489287 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs" Nov 03 09:16:22.776537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.773879 5015 generic.go:356] "Generic (PLEG): container finished" podID="56530cb6-7d91-469a-96cb-4049f1ec5e5a" containerID="44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec" exitCode=0 Nov 03 09:16:22.776537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.774139 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" event={"ID":"56530cb6-7d91-469a-96cb-4049f1ec5e5a","Type":"ContainerDied","Data":"44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec"} Nov 03 09:16:22.776537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.774194 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" event={"ID":"56530cb6-7d91-469a-96cb-4049f1ec5e5a","Type":"ContainerDied","Data":"02346db149fec693a74b92222d80781f5ae67e52a1476da93506ee6ee5636489"} Nov 03 09:16:22.776537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.774300 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68" Nov 03 09:16:22.861513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.857660 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs"] Nov 03 09:16:22.914393 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.913982 5015 scope.go:122] "RemoveContainer" containerID="229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd" Nov 03 09:16:22.921543 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:16:22.914676 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd\": container with ID starting with 229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd not found: ID does not exist" containerID="229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd" Nov 03 09:16:22.921543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.914714 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd"} err="failed to get container status \"229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd\": rpc error: code = NotFound desc = could not find container \"229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd\": container with ID starting with 229fd617b56415a92da791ef5393657589e8e952f57ddd9355d9a36e401d09cd not found: ID does not exist" Nov 03 09:16:22.921543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.914739 5015 scope.go:122] "RemoveContainer" containerID="44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec" Nov 03 09:16:22.936948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:22.936916 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c32d41-57d57f9ccf-cp5hs"] Nov 03 09:16:23.047747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:23.044171 5015 scope.go:122] "RemoveContainer" containerID="f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517" Nov 03 09:16:23.111673 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:23.111599 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68"] Nov 03 09:16:23.137510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:23.137069 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c32d41-7cb4b8df8-ftb68"] Nov 03 09:16:23.285869 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:23.285837 5015 scope.go:122] "RemoveContainer" containerID="44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec" Nov 03 09:16:23.312391 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:16:23.299716 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec\": container with ID starting with 44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec not found: ID does not exist" containerID="44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec" Nov 03 09:16:23.312391 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:23.299766 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec"} err="failed to get container status \"44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec\": rpc error: code = NotFound desc = could not find container \"44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec\": container with ID starting with 44618ecd863edea8f334e291594abf442897d9b951d1113888d6dd5d7f50d3ec not found: ID does not exist" Nov 03 09:16:23.312391 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:23.299795 5015 scope.go:122] "RemoveContainer" containerID="f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517" Nov 03 09:16:23.324560 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:16:23.324127 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517\": container with ID starting with f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517 not found: ID does not exist" containerID="f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517" Nov 03 09:16:23.324560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:23.324179 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517"} err="failed to get container status \"f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517\": rpc error: code = NotFound desc = could not find container \"f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517\": container with ID starting with f9b57651e4cb54b81c14c24ce70c04f4917e9cc4438fefa560cbc45d0b194517 not found: ID does not exist" Nov 03 09:16:23.358503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:23.358048 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56530cb6-7d91-469a-96cb-4049f1ec5e5a" path="/var/lib/kubelet/pods/56530cb6-7d91-469a-96cb-4049f1ec5e5a/volumes" Nov 03 09:16:23.372550 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:23.358752 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a758c014-84c7-425e-b9cb-5ea42c09625d" path="/var/lib/kubelet/pods/a758c014-84c7-425e-b9cb-5ea42c09625d/volumes" Nov 03 09:16:23.477511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:23.469581 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:16:24.013511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:24.012573 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 09:16:28.364113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:28.364065 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7"] Nov 03 09:16:28.364652 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:28.364340 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" containerID="cri-o://97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0" gracePeriod=30 Nov 03 09:16:30.279717 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.279676 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx"] Nov 03 09:16:30.280689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.280666 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a758c014-84c7-425e-b9cb-5ea42c09625d" containerName="payload-pre-processing" Nov 03 09:16:30.280751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.280692 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a758c014-84c7-425e-b9cb-5ea42c09625d" containerName="payload-pre-processing" Nov 03 09:16:30.280751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.280724 5015 cpu_manager.go:414] "containerMap: removing container" podUID="56530cb6-7d91-469a-96cb-4049f1ec5e5a" containerName="payload-processing" Nov 03 09:16:30.280751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.280732 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56530cb6-7d91-469a-96cb-4049f1ec5e5a" containerName="payload-processing" Nov 03 09:16:30.280751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.280745 5015 cpu_manager.go:414] "containerMap: removing container" podUID="56530cb6-7d91-469a-96cb-4049f1ec5e5a" containerName="payload-processing" Nov 03 09:16:30.280883 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.280752 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56530cb6-7d91-469a-96cb-4049f1ec5e5a" containerName="payload-processing" Nov 03 09:16:30.280957 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.280940 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="56530cb6-7d91-469a-96cb-4049f1ec5e5a" containerName="payload-processing" Nov 03 09:16:30.280995 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.280967 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="56530cb6-7d91-469a-96cb-4049f1ec5e5a" containerName="payload-processing" Nov 03 09:16:30.280995 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.280991 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a758c014-84c7-425e-b9cb-5ea42c09625d" containerName="payload-pre-processing" Nov 03 09:16:30.295919 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.295884 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.351404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.344065 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-istio-token\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.351404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.344149 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-socket\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.351404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.344196 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-credential-socket\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.351404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.344242 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3c1ad9-a089-4846-9662-db73cd849332-istiod-ca-cert\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.351404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.344267 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zptt8\" (UniqueName: \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-kube-api-access-zptt8\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.351404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.344322 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-data\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.351404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.344350 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-certs\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.351404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.344396 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-envoy\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.351404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.344423 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3c1ad9-a089-4846-9662-db73cd849332-istio-podinfo\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.353836 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.353806 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-1ce7a7-openshift-default-dockercfg-tb8wq\"" Nov 03 09:16:30.411650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.411604 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx"] Nov 03 09:16:30.460879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.460839 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-data\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.461083 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.460892 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-certs\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.461083 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.460939 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-envoy\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.461083 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.460969 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3c1ad9-a089-4846-9662-db73cd849332-istio-podinfo\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.461263 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.461151 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-istio-token\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.461263 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.461202 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-socket\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.461263 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.461237 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-credential-socket\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.461433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.461279 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3c1ad9-a089-4846-9662-db73cd849332-istiod-ca-cert\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.461433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.461304 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zptt8\" (UniqueName: \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-kube-api-access-zptt8\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.462581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.462008 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-data\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.463616 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.463583 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-certs\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.467272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.467241 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-socket\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.468807 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.468777 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-credential-socket\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.470042 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.469433 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3c1ad9-a089-4846-9662-db73cd849332-istiod-ca-cert\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.470042 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.469658 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-envoy\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.470781 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.470420 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3c1ad9-a089-4846-9662-db73cd849332-istio-podinfo\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.525204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.524309 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zptt8\" (UniqueName: \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-kube-api-access-zptt8\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.530363 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.530284 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-istio-token\") pod \"e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:30.645736 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:30.645691 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:31.123843 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:16:31.123813 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3c1ad9_a089_4846_9662_db73cd849332.slice/crio-366439aa7d4be0b29e929558809791e5681d519d0e167508314d49c66e25f33c WatchSource:0}: Error finding container 366439aa7d4be0b29e929558809791e5681d519d0e167508314d49c66e25f33c: Status 404 returned error can't find the container with id 366439aa7d4be0b29e929558809791e5681d519d0e167508314d49c66e25f33c Nov 03 09:16:31.124097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.124074 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx"] Nov 03 09:16:31.126566 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.126532 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:31.126643 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.126602 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:31.126643 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.126635 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:31.963042 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.963008 5015 generic.go:356] "Generic (PLEG): container finished" podID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerID="601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f" exitCode=0 Nov 03 09:16:31.963611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.963079 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" event={"ID":"37ebc178-9bdf-4b0d-9d34-13b9fc475477","Type":"ContainerDied","Data":"601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f"} Nov 03 09:16:31.963710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.963631 5015 scope.go:122] "RemoveContainer" containerID="601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f" Nov 03 09:16:31.965580 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.965543 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:31.965681 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.965637 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:31.965727 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.965684 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:31.967578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.967231 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" event={"ID":"7c3c1ad9-a089-4846-9662-db73cd849332","Type":"ContainerStarted","Data":"dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885"} Nov 03 09:16:31.967578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:31.967286 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" event={"ID":"7c3c1ad9-a089-4846-9662-db73cd849332","Type":"ContainerStarted","Data":"366439aa7d4be0b29e929558809791e5681d519d0e167508314d49c66e25f33c"} Nov 03 09:16:32.205882 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:32.205769 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" podStartSLOduration=2.205755695 podStartE2EDuration="2.205755695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:32.203434434 +0000 UTC m=+2225.128376652" watchObservedRunningTime="2026-11-03 09:16:32.205755695 +0000 UTC m=+2225.130697911" Nov 03 09:16:32.646677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:32.646639 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:32.653269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:32.653244 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:32.977003 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:32.976972 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" event={"ID":"37ebc178-9bdf-4b0d-9d34-13b9fc475477","Type":"ContainerStarted","Data":"d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599"} Nov 03 09:16:32.977561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:32.977186 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:32.978294 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:32.978266 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:16:33.722899 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.722872 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:16:33.814403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814370 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-kube-api-access-c57ds\" (UniqueName: \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-kube-api-access-c57ds\") pod \"665332b4-9c0b-4a43-ab38-2050abc0521c\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " Nov 03 09:16:33.814403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814407 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-socket\") pod \"665332b4-9c0b-4a43-ab38-2050abc0521c\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " Nov 03 09:16:33.814683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814446 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-podinfo\") pod \"665332b4-9c0b-4a43-ab38-2050abc0521c\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " Nov 03 09:16:33.814683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814504 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/665332b4-9c0b-4a43-ab38-2050abc0521c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665332b4-9c0b-4a43-ab38-2050abc0521c-istiod-ca-cert\") pod \"665332b4-9c0b-4a43-ab38-2050abc0521c\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " Nov 03 09:16:33.814683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814537 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-certs\") pod \"665332b4-9c0b-4a43-ab38-2050abc0521c\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " Nov 03 09:16:33.814683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814581 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-token\" (UniqueName: \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-token\") pod \"665332b4-9c0b-4a43-ab38-2050abc0521c\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " Nov 03 09:16:33.814683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814622 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-data\") pod \"665332b4-9c0b-4a43-ab38-2050abc0521c\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " Nov 03 09:16:33.814949 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814712 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-socket" pod "665332b4-9c0b-4a43-ab38-2050abc0521c" (UID: "665332b4-9c0b-4a43-ab38-2050abc0521c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:33.814949 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814769 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-envoy\") pod \"665332b4-9c0b-4a43-ab38-2050abc0521c\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " Nov 03 09:16:33.814949 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814817 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-credential-socket\") pod \"665332b4-9c0b-4a43-ab38-2050abc0521c\" (UID: \"665332b4-9c0b-4a43-ab38-2050abc0521c\") " Nov 03 09:16:33.814949 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814888 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-certs" pod "665332b4-9c0b-4a43-ab38-2050abc0521c" (UID: "665332b4-9c0b-4a43-ab38-2050abc0521c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:33.815158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.814958 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/665332b4-9c0b-4a43-ab38-2050abc0521c-istiod-ca-cert" pod "665332b4-9c0b-4a43-ab38-2050abc0521c" (UID: "665332b4-9c0b-4a43-ab38-2050abc0521c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:16:33.815158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.815045 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-data" pod "665332b4-9c0b-4a43-ab38-2050abc0521c" (UID: "665332b4-9c0b-4a43-ab38-2050abc0521c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:33.815271 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.815209 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-credential-socket" pod "665332b4-9c0b-4a43-ab38-2050abc0521c" (UID: "665332b4-9c0b-4a43-ab38-2050abc0521c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:33.815565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.815496 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:33.815565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.815528 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:33.815565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.815543 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665332b4-9c0b-4a43-ab38-2050abc0521c-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:33.815565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.815556 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:33.815565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.815571 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:33.817066 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.817036 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-podinfo" pod "665332b4-9c0b-4a43-ab38-2050abc0521c" (UID: "665332b4-9c0b-4a43-ab38-2050abc0521c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:16:33.817222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.817154 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-envoy" pod "665332b4-9c0b-4a43-ab38-2050abc0521c" (UID: "665332b4-9c0b-4a43-ab38-2050abc0521c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:33.817222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.817168 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-kube-api-access-c57ds" pod "665332b4-9c0b-4a43-ab38-2050abc0521c" (UID: "665332b4-9c0b-4a43-ab38-2050abc0521c"). InnerVolumeSpecName "kube-api-access-c57ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:33.817423 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.817408 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-token" pod "665332b4-9c0b-4a43-ab38-2050abc0521c" (UID: "665332b4-9c0b-4a43-ab38-2050abc0521c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:33.894715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.894450 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:16:33.894715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.894508 5015 patch_prober.go:29] interesting pod/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 09:16:33.894715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.894564 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 09:16:33.916740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.916699 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c57ds\" (UniqueName: \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-kube-api-access-c57ds\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:33.916740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.916734 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:33.916933 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.916750 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:33.916933 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.916762 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665332b4-9c0b-4a43-ab38-2050abc0521c-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:33.989454 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.989418 5015 generic.go:356] "Generic (PLEG): container finished" podID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerID="97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0" exitCode=0 Nov 03 09:16:33.989984 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.989521 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" Nov 03 09:16:33.989984 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.989516 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" event={"ID":"665332b4-9c0b-4a43-ab38-2050abc0521c","Type":"ContainerDied","Data":"97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0"} Nov 03 09:16:33.989984 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.989606 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7" event={"ID":"665332b4-9c0b-4a43-ab38-2050abc0521c","Type":"ContainerDied","Data":"86f71f5ec9ffbc0ca25ed922871248aafb0413afaabd18aee55131d356443251"} Nov 03 09:16:33.989984 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:33.989639 5015 scope.go:122] "RemoveContainer" containerID="97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0" Nov 03 09:16:34.002235 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:34.002212 5015 scope.go:122] "RemoveContainer" containerID="8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3" Nov 03 09:16:34.012078 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:34.012058 5015 scope.go:122] "RemoveContainer" containerID="97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0" Nov 03 09:16:34.012341 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:16:34.012321 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0\": container with ID starting with 97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0 not found: ID does not exist" containerID="97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0" Nov 03 09:16:34.012398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:34.012349 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0"} err="failed to get container status \"97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0\": rpc error: code = NotFound desc = could not find container \"97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0\": container with ID starting with 97f81de407aa5fecb0c86283b6bef5004354c6d91b3ec4570caba88e38ba8ef0 not found: ID does not exist" Nov 03 09:16:34.012398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:34.012367 5015 scope.go:122] "RemoveContainer" containerID="8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3" Nov 03 09:16:34.012763 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:16:34.012743 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3\": container with ID starting with 8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3 not found: ID does not exist" containerID="8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3" Nov 03 09:16:34.012838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:34.012774 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3"} err="failed to get container status \"8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3\": rpc error: code = NotFound desc = could not find container \"8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3\": container with ID starting with 8b1111fca97aaace77d02ef4748b7b6badd829210cff1f203837110d72029df3 not found: ID does not exist" Nov 03 09:16:34.192364 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:34.192327 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7"] Nov 03 09:16:34.203963 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:34.203928 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c32d41-openshift-default-5c6fb896b7-rxtv7"] Nov 03 09:16:34.894203 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:34.894165 5015 patch_prober.go:29] interesting pod/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 09:16:34.894404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:34.894218 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 09:16:35.205921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:35.205840 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" path="/var/lib/kubelet/pods/665332b4-9c0b-4a43-ab38-2050abc0521c/volumes" Nov 03 09:16:35.895047 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:35.895012 5015 patch_prober.go:29] interesting pod/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 09:16:35.895239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:35.895068 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 09:16:36.895239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:36.895200 5015 patch_prober.go:29] interesting pod/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 09:16:36.895679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:36.895269 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 09:16:37.894331 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:37.894293 5015 patch_prober.go:29] interesting pod/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 09:16:37.894534 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:37.894365 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 09:16:38.894931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:38.894892 5015 patch_prober.go:29] interesting pod/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 09:16:38.895588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:38.895552 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 09:16:39.922753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:39.916766 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:16:39.922753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:39.918248 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:16:39.922753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:39.920684 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:16:46.903698 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:16:46.903649 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:16:53.857783 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.857743 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws"] Nov 03 09:16:53.858267 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.857941 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" containerID="cri-o://ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f" gracePeriod=30 Nov 03 09:16:53.908494 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.908450 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs"] Nov 03 09:16:53.909129 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.909111 5015 cpu_manager.go:414] "containerMap: removing container" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" Nov 03 09:16:53.909197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.909131 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" Nov 03 09:16:53.909197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.909153 5015 cpu_manager.go:414] "containerMap: removing container" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" Nov 03 09:16:53.909197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.909158 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" Nov 03 09:16:53.909296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.909278 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" Nov 03 09:16:53.909296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.909289 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="665332b4-9c0b-4a43-ab38-2050abc0521c" containerName="istio-proxy" Nov 03 09:16:53.913005 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.912982 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:53.946205 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.946172 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvp7k\" (UniqueName: \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-kube-api-access-fvp7k\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:53.946377 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.946235 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:53.946377 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.946261 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:53.946377 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.946292 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:53.946377 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.946338 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:53.946625 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.946391 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:53.946625 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.946434 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:53.946625 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.946506 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:53.946780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:53.946630 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.030763 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.030730 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs"] Nov 03 09:16:54.050689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.050648 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.050887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.050736 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.050887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.050864 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.051008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.050906 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvp7k\" (UniqueName: \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-kube-api-access-fvp7k\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.051008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.050957 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.051008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.050982 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.052243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.051012 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.052243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.051284 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.052243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.051364 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.052243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.051629 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.052243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.051888 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.052243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.051909 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.052243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.052004 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.052957 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.052934 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.055753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.055731 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.057895 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.057873 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.114049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.113972 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.114214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.114051 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvp7k\" (UniqueName: \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-kube-api-access-fvp7k\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.229329 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.229296 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:54.669022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.668986 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs"] Nov 03 09:16:54.669439 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:16:54.669410 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbc2cb6_1360_456b_8c90_24f83fbdebd9.slice/crio-85e84fbb1fb29f090790e892477e5871244d975ff2e34b650dffc9a701f6adb8 WatchSource:0}: Error finding container 85e84fbb1fb29f090790e892477e5871244d975ff2e34b650dffc9a701f6adb8: Status 404 returned error can't find the container with id 85e84fbb1fb29f090790e892477e5871244d975ff2e34b650dffc9a701f6adb8 Nov 03 09:16:54.672528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.672448 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:54.672618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.672579 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:54.672663 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:54.672625 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:16:55.213900 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:55.213862 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" event={"ID":"dfbc2cb6-1360-456b-8c90-24f83fbdebd9","Type":"ContainerStarted","Data":"ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62"} Nov 03 09:16:55.214293 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:55.213921 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" event={"ID":"dfbc2cb6-1360-456b-8c90-24f83fbdebd9","Type":"ContainerStarted","Data":"85e84fbb1fb29f090790e892477e5871244d975ff2e34b650dffc9a701f6adb8"} Nov 03 09:16:55.229699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:55.229671 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:16:55.231569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:55.231549 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 09:16:55.231634 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:55.231594 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 09:16:55.345509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:55.336727 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" podStartSLOduration=2.336711451 podStartE2EDuration="2.336711451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:55.33567708 +0000 UTC m=+2248.260619321" watchObservedRunningTime="2026-11-03 09:16:55.336711451 +0000 UTC m=+2248.261653669" Nov 03 09:16:56.230393 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:56.230354 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 09:16:56.230801 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:56.230407 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 09:16:57.230764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:57.230727 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 09:16:57.231175 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:57.230792 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 09:16:57.672194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:57.672102 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:16:57.672194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:57.672165 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:16:57.890020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:57.889891 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt"] Nov 03 09:16:57.890226 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:57.890157 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" containerID="cri-o://f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c" gracePeriod=30 Nov 03 09:16:58.229986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:58.229947 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 09:16:58.230148 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:58.230000 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 09:16:59.206964 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.206937 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:16:59.230295 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.230255 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 09:16:59.230451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.230313 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 09:16:59.272544 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.272503 5015 generic.go:356] "Generic (PLEG): container finished" podID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerID="ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f" exitCode=0 Nov 03 09:16:59.272759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.272643 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" event={"ID":"c6cff466-5838-4ed8-9124-09bdc768a0ca","Type":"ContainerDied","Data":"ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f"} Nov 03 09:16:59.272759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.272696 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" event={"ID":"c6cff466-5838-4ed8-9124-09bdc768a0ca","Type":"ContainerDied","Data":"bdbad6126a74ad8ed6805ad9c7cc54d76debe65dc081068f0d22074e6e58bc16"} Nov 03 09:16:59.272759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.272727 5015 scope.go:122] "RemoveContainer" containerID="ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f" Nov 03 09:16:59.272933 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.272919 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws" Nov 03 09:16:59.287365 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.287340 5015 scope.go:122] "RemoveContainer" containerID="ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f" Nov 03 09:16:59.287684 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:16:59.287659 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f\": container with ID starting with ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f not found: ID does not exist" containerID="ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f" Nov 03 09:16:59.287744 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.287702 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f"} err="failed to get container status \"ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f\": rpc error: code = NotFound desc = could not find container \"ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f\": container with ID starting with ecd3b76cc3fd0f765f819df745b6516218509537e62b71f3ed06985af9ed725f not found: ID does not exist" Nov 03 09:16:59.333984 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.333955 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6cff466-5838-4ed8-9124-09bdc768a0ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6cff466-5838-4ed8-9124-09bdc768a0ca-istiod-ca-cert\") pod \"c6cff466-5838-4ed8-9124-09bdc768a0ca\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " Nov 03 09:16:59.334144 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334125 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-socket\") pod \"c6cff466-5838-4ed8-9124-09bdc768a0ca\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " Nov 03 09:16:59.334193 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334165 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-certs\") pod \"c6cff466-5838-4ed8-9124-09bdc768a0ca\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " Nov 03 09:16:59.334264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334232 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-podinfo\") pod \"c6cff466-5838-4ed8-9124-09bdc768a0ca\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " Nov 03 09:16:59.334307 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334286 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-token\" (UniqueName: \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-token\") pod \"c6cff466-5838-4ed8-9124-09bdc768a0ca\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " Nov 03 09:16:59.334361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334344 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-envoy\") pod \"c6cff466-5838-4ed8-9124-09bdc768a0ca\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " Nov 03 09:16:59.334416 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334389 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-certs" pod "c6cff466-5838-4ed8-9124-09bdc768a0ca" (UID: "c6cff466-5838-4ed8-9124-09bdc768a0ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:59.334416 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334398 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-data\") pod \"c6cff466-5838-4ed8-9124-09bdc768a0ca\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " Nov 03 09:16:59.334533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334411 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-socket" pod "c6cff466-5838-4ed8-9124-09bdc768a0ca" (UID: "c6cff466-5838-4ed8-9124-09bdc768a0ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:59.334579 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334528 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-kube-api-access-5l5s7\" (UniqueName: \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-kube-api-access-5l5s7\") pod \"c6cff466-5838-4ed8-9124-09bdc768a0ca\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " Nov 03 09:16:59.334579 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334566 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-credential-socket\") pod \"c6cff466-5838-4ed8-9124-09bdc768a0ca\" (UID: \"c6cff466-5838-4ed8-9124-09bdc768a0ca\") " Nov 03 09:16:59.334683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334649 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-data" pod "c6cff466-5838-4ed8-9124-09bdc768a0ca" (UID: "c6cff466-5838-4ed8-9124-09bdc768a0ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:59.334938 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.334893 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6cff466-5838-4ed8-9124-09bdc768a0ca-istiod-ca-cert" pod "c6cff466-5838-4ed8-9124-09bdc768a0ca" (UID: "c6cff466-5838-4ed8-9124-09bdc768a0ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:16:59.335146 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.335030 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-credential-socket" pod "c6cff466-5838-4ed8-9124-09bdc768a0ca" (UID: "c6cff466-5838-4ed8-9124-09bdc768a0ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:59.335272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.335251 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:59.335318 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.335278 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:59.335318 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.335291 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:59.335318 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.335304 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:59.335318 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.335315 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6cff466-5838-4ed8-9124-09bdc768a0ca-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:59.358577 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.358545 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-token" pod "c6cff466-5838-4ed8-9124-09bdc768a0ca" (UID: "c6cff466-5838-4ed8-9124-09bdc768a0ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:59.361546 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.359587 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-envoy" pod "c6cff466-5838-4ed8-9124-09bdc768a0ca" (UID: "c6cff466-5838-4ed8-9124-09bdc768a0ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:59.364714 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.364660 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-kube-api-access-5l5s7" pod "c6cff466-5838-4ed8-9124-09bdc768a0ca" (UID: "c6cff466-5838-4ed8-9124-09bdc768a0ca"). InnerVolumeSpecName "kube-api-access-5l5s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:59.370508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.369803 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-podinfo" pod "c6cff466-5838-4ed8-9124-09bdc768a0ca" (UID: "c6cff466-5838-4ed8-9124-09bdc768a0ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:16:59.461334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.437164 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:59.461334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.437202 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:59.461334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.437218 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6cff466-5838-4ed8-9124-09bdc768a0ca-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:59.461334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.437232 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l5s7\" (UniqueName: \"kubernetes.io/projected/c6cff466-5838-4ed8-9124-09bdc768a0ca-kube-api-access-5l5s7\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:59.702505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.702440 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws"] Nov 03 09:16:59.707361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:16:59.707335 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-v6bws"] Nov 03 09:17:00.230570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:00.230526 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 09:17:00.231068 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:00.230603 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 09:17:01.235511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:01.234587 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 09:17:01.235511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:01.234661 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 09:17:01.261507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:01.261083 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" path="/var/lib/kubelet/pods/c6cff466-5838-4ed8-9124-09bdc768a0ca/volumes" Nov 03 09:17:02.234397 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:02.234365 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:17:02.234688 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:02.234668 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:17:02.235313 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:02.235294 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:17:02.788401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:02.788368 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7"] Nov 03 09:17:02.897336 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:02.897302 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77"] Nov 03 09:17:03.191838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.191810 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 09:17:03.327729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.326109 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-kube-api-access-fnjlp\" (UniqueName: \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-kube-api-access-fnjlp\") pod \"66f961ac-69cf-407a-af9b-07611b20726c\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " Nov 03 09:17:03.327729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.326202 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-credential-socket\") pod \"66f961ac-69cf-407a-af9b-07611b20726c\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " Nov 03 09:17:03.327729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.326237 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-istio-token\" (UniqueName: \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-istio-token\") pod \"66f961ac-69cf-407a-af9b-07611b20726c\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " Nov 03 09:17:03.327729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.326271 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f961ac-69cf-407a-af9b-07611b20726c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f961ac-69cf-407a-af9b-07611b20726c-istiod-ca-cert\") pod \"66f961ac-69cf-407a-af9b-07611b20726c\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " Nov 03 09:17:03.327729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.326341 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-socket\") pod \"66f961ac-69cf-407a-af9b-07611b20726c\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " Nov 03 09:17:03.327729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.326398 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-data\") pod \"66f961ac-69cf-407a-af9b-07611b20726c\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " Nov 03 09:17:03.327729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.326564 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66f961ac-69cf-407a-af9b-07611b20726c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f961ac-69cf-407a-af9b-07611b20726c-istio-podinfo\") pod \"66f961ac-69cf-407a-af9b-07611b20726c\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " Nov 03 09:17:03.327729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.326631 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-certs\") pod \"66f961ac-69cf-407a-af9b-07611b20726c\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " Nov 03 09:17:03.327729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.326673 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-envoy\") pod \"66f961ac-69cf-407a-af9b-07611b20726c\" (UID: \"66f961ac-69cf-407a-af9b-07611b20726c\") " Nov 03 09:17:03.334890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.334108 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-credential-socket" pod "66f961ac-69cf-407a-af9b-07611b20726c" (UID: "66f961ac-69cf-407a-af9b-07611b20726c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:03.334890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.334618 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-data" pod "66f961ac-69cf-407a-af9b-07611b20726c" (UID: "66f961ac-69cf-407a-af9b-07611b20726c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:03.334890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.334869 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-socket" pod "66f961ac-69cf-407a-af9b-07611b20726c" (UID: "66f961ac-69cf-407a-af9b-07611b20726c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:03.335281 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.335071 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-certs" pod "66f961ac-69cf-407a-af9b-07611b20726c" (UID: "66f961ac-69cf-407a-af9b-07611b20726c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:03.335530 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.335507 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f961ac-69cf-407a-af9b-07611b20726c-istiod-ca-cert" pod "66f961ac-69cf-407a-af9b-07611b20726c" (UID: "66f961ac-69cf-407a-af9b-07611b20726c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:17:03.345591 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.345555 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-envoy" pod "66f961ac-69cf-407a-af9b-07611b20726c" (UID: "66f961ac-69cf-407a-af9b-07611b20726c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:03.345726 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.345668 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-istio-token" pod "66f961ac-69cf-407a-af9b-07611b20726c" (UID: "66f961ac-69cf-407a-af9b-07611b20726c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:03.345805 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.345724 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-kube-api-access-fnjlp" pod "66f961ac-69cf-407a-af9b-07611b20726c" (UID: "66f961ac-69cf-407a-af9b-07611b20726c"). InnerVolumeSpecName "kube-api-access-fnjlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:03.346152 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.346129 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66f961ac-69cf-407a-af9b-07611b20726c-istio-podinfo" pod "66f961ac-69cf-407a-af9b-07611b20726c" (UID: "66f961ac-69cf-407a-af9b-07611b20726c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:17:03.359046 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.359003 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw"] Nov 03 09:17:03.359563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.359300 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" containerID="cri-o://9810235e28f946dd422a893006a60cd65212c61893614fec96c8aca6151de1db" gracePeriod=30 Nov 03 09:17:03.367167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.367141 5015 generic.go:356] "Generic (PLEG): container finished" podID="66f961ac-69cf-407a-af9b-07611b20726c" containerID="f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c" exitCode=0 Nov 03 09:17:03.367794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.367772 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" event={"ID":"66f961ac-69cf-407a-af9b-07611b20726c","Type":"ContainerDied","Data":"f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c"} Nov 03 09:17:03.368028 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.368013 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" event={"ID":"66f961ac-69cf-407a-af9b-07611b20726c","Type":"ContainerDied","Data":"fd76f7f8f59b1c3d4f7845b1d1f7c2a5f8091f6034bde1a9901bcfe24259683e"} Nov 03 09:17:03.368158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.368146 5015 scope.go:122] "RemoveContainer" containerID="f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c" Nov 03 09:17:03.370954 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.370941 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt" Nov 03 09:17:03.395060 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.395042 5015 scope.go:122] "RemoveContainer" containerID="ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781" Nov 03 09:17:03.405886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.405865 5015 scope.go:122] "RemoveContainer" containerID="f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c" Nov 03 09:17:03.406180 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:03.406160 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c\": container with ID starting with f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c not found: ID does not exist" containerID="f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c" Nov 03 09:17:03.406256 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.406194 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c"} err="failed to get container status \"f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c\": rpc error: code = NotFound desc = could not find container \"f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c\": container with ID starting with f5b51bd58a21d6a61d9679d714b2ae082f06e92c150f2455798bdeb241f8740c not found: ID does not exist" Nov 03 09:17:03.406256 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.406220 5015 scope.go:122] "RemoveContainer" containerID="ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781" Nov 03 09:17:03.406527 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:03.406509 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781\": container with ID starting with ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781 not found: ID does not exist" containerID="ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781" Nov 03 09:17:03.406586 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.406533 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781"} err="failed to get container status \"ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781\": rpc error: code = NotFound desc = could not find container \"ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781\": container with ID starting with ed3c3e2debd7f701a08f62f3ea2ed838f1304b7e065eb83ac897da822284e781 not found: ID does not exist" Nov 03 09:17:03.428990 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.428964 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:03.428990 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.428992 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:03.429110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.429002 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f961ac-69cf-407a-af9b-07611b20726c-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:03.429110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.429011 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:03.429110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.429019 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:03.429110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.429028 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnjlp\" (UniqueName: \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-kube-api-access-fnjlp\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:03.429110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.429037 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f961ac-69cf-407a-af9b-07611b20726c-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:03.429110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.429049 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f961ac-69cf-407a-af9b-07611b20726c-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:03.429110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.429062 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f961ac-69cf-407a-af9b-07611b20726c-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:03.628753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.628674 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt"] Nov 03 09:17:03.635553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:03.635526 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c89vt"] Nov 03 09:17:05.207661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:05.207627 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f961ac-69cf-407a-af9b-07611b20726c" path="/var/lib/kubelet/pods/66f961ac-69cf-407a-af9b-07611b20726c/volumes" Nov 03 09:17:06.577413 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:06.577380 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84"] Nov 03 09:17:06.577866 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:06.577624 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" containerName="istio-proxy" containerID="cri-o://a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966" gracePeriod=30 Nov 03 09:17:07.364639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.364604 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj"] Nov 03 09:17:07.364834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.364804 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" podUID="f02a39f0-92d4-4963-8be6-18dbe4da520e" containerName="maas-api" containerID="cri-o://71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88" gracePeriod=30 Nov 03 09:17:07.702018 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.701960 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:17:07.707746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.707708 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-metrics-certs\") pod \"f02a39f0-92d4-4963-8be6-18dbe4da520e\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " Nov 03 09:17:07.708493 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.708452 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02a39f0-92d4-4963-8be6-18dbe4da520e-kube-api-access-n7blx\" (UniqueName: \"kubernetes.io/projected/f02a39f0-92d4-4963-8be6-18dbe4da520e-kube-api-access-n7blx\") pod \"f02a39f0-92d4-4963-8be6-18dbe4da520e\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " Nov 03 09:17:07.708776 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.708757 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-maas-api-tls\") pod \"f02a39f0-92d4-4963-8be6-18dbe4da520e\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " Nov 03 09:17:07.708972 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.708942 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02a39f0-92d4-4963-8be6-18dbe4da520e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02a39f0-92d4-4963-8be6-18dbe4da520e-openshift-service-ca\") pod \"f02a39f0-92d4-4963-8be6-18dbe4da520e\" (UID: \"f02a39f0-92d4-4963-8be6-18dbe4da520e\") " Nov 03 09:17:07.710557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.709446 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02a39f0-92d4-4963-8be6-18dbe4da520e-openshift-service-ca" pod "f02a39f0-92d4-4963-8be6-18dbe4da520e" (UID: "f02a39f0-92d4-4963-8be6-18dbe4da520e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:17:07.732326 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.732255 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02a39f0-92d4-4963-8be6-18dbe4da520e-kube-api-access-n7blx" pod "f02a39f0-92d4-4963-8be6-18dbe4da520e" (UID: "f02a39f0-92d4-4963-8be6-18dbe4da520e"). InnerVolumeSpecName "kube-api-access-n7blx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:07.732661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.732618 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-metrics-certs" pod "f02a39f0-92d4-4963-8be6-18dbe4da520e" (UID: "f02a39f0-92d4-4963-8be6-18dbe4da520e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:17:07.732822 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.732721 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-maas-api-tls" pod "f02a39f0-92d4-4963-8be6-18dbe4da520e" (UID: "f02a39f0-92d4-4963-8be6-18dbe4da520e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:17:07.790896 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.789711 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" podUID="3449b460-2c34-428a-a2c7-71b5dba3b16a" containerName="payload-processing" containerID="cri-o://5dfbb7ceea0a66fd725f1fa82c31dfa3e2a2944d4cc6468bd26802472dc29065" gracePeriod=25 Nov 03 09:17:07.812195 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.812089 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7blx\" (UniqueName: \"kubernetes.io/projected/f02a39f0-92d4-4963-8be6-18dbe4da520e-kube-api-access-n7blx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:07.812195 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.812143 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:07.812195 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.812158 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02a39f0-92d4-4963-8be6-18dbe4da520e-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:07.812195 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.812171 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f02a39f0-92d4-4963-8be6-18dbe4da520e-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:07.898430 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:07.897659 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" podUID="3f67346f-7c34-460d-898e-30dd07e22f12" containerName="payload-pre-processing" containerID="cri-o://139c81dbe0a097974500d993503c57e89ac7cebd67b521c89122ff28c1040c4b" gracePeriod=25 Nov 03 09:17:08.500596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.500549 5015 generic.go:356] "Generic (PLEG): container finished" podID="f02a39f0-92d4-4963-8be6-18dbe4da520e" containerID="71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88" exitCode=0 Nov 03 09:17:08.501092 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.500824 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" event={"ID":"f02a39f0-92d4-4963-8be6-18dbe4da520e","Type":"ContainerDied","Data":"71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88"} Nov 03 09:17:08.501304 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.501281 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" event={"ID":"f02a39f0-92d4-4963-8be6-18dbe4da520e","Type":"ContainerDied","Data":"bb3c1d9f7726b95295a4cd5546b89a23658a401f9a7ea9679a2a08b4d1fd530f"} Nov 03 09:17:08.501433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.501421 5015 scope.go:122] "RemoveContainer" containerID="71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88" Nov 03 09:17:08.501665 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.500906 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj" Nov 03 09:17:08.506047 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.505736 5015 generic.go:356] "Generic (PLEG): container finished" podID="3449b460-2c34-428a-a2c7-71b5dba3b16a" containerID="5dfbb7ceea0a66fd725f1fa82c31dfa3e2a2944d4cc6468bd26802472dc29065" exitCode=0 Nov 03 09:17:08.506047 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.505969 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" event={"ID":"3449b460-2c34-428a-a2c7-71b5dba3b16a","Type":"ContainerDied","Data":"5dfbb7ceea0a66fd725f1fa82c31dfa3e2a2944d4cc6468bd26802472dc29065"} Nov 03 09:17:08.521456 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.521294 5015 generic.go:356] "Generic (PLEG): container finished" podID="3f67346f-7c34-460d-898e-30dd07e22f12" containerID="139c81dbe0a097974500d993503c57e89ac7cebd67b521c89122ff28c1040c4b" exitCode=0 Nov 03 09:17:08.521456 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.521352 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" event={"ID":"3f67346f-7c34-460d-898e-30dd07e22f12","Type":"ContainerDied","Data":"139c81dbe0a097974500d993503c57e89ac7cebd67b521c89122ff28c1040c4b"} Nov 03 09:17:08.541362 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:17:08.541339 5015 logging.go:55] [core] [Channel #4598 SubChannel #4599]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.152:50052", ServerName: "10.132.0.152:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.152:50052: connect: connection refused" Nov 03 09:17:08.580008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.579968 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:17:08.683500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.681090 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3449b460-2c34-428a-a2c7-71b5dba3b16a-kube-api-access-rjmpt\" (UniqueName: \"kubernetes.io/projected/3449b460-2c34-428a-a2c7-71b5dba3b16a-kube-api-access-rjmpt\") pod \"3449b460-2c34-428a-a2c7-71b5dba3b16a\" (UID: \"3449b460-2c34-428a-a2c7-71b5dba3b16a\") " Nov 03 09:17:08.683500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.681344 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3449b460-2c34-428a-a2c7-71b5dba3b16a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3449b460-2c34-428a-a2c7-71b5dba3b16a-plugins-config-volume\") pod \"3449b460-2c34-428a-a2c7-71b5dba3b16a\" (UID: \"3449b460-2c34-428a-a2c7-71b5dba3b16a\") " Nov 03 09:17:08.683500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.682744 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3449b460-2c34-428a-a2c7-71b5dba3b16a-plugins-config-volume" pod "3449b460-2c34-428a-a2c7-71b5dba3b16a" (UID: "3449b460-2c34-428a-a2c7-71b5dba3b16a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:17:08.715461 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.715370 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj"] Nov 03 09:17:08.732960 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.731658 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3449b460-2c34-428a-a2c7-71b5dba3b16a-kube-api-access-rjmpt" pod "3449b460-2c34-428a-a2c7-71b5dba3b16a" (UID: "3449b460-2c34-428a-a2c7-71b5dba3b16a"). InnerVolumeSpecName "kube-api-access-rjmpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:08.751222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.750763 5015 scope.go:122] "RemoveContainer" containerID="71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88" Nov 03 09:17:08.752140 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:08.752058 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88\": container with ID starting with 71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88 not found: ID does not exist" containerID="71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88" Nov 03 09:17:08.752140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.752100 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88"} err="failed to get container status \"71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88\": rpc error: code = NotFound desc = could not find container \"71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88\": container with ID starting with 71e4008cfcbe5a450acc75e654f602644b29dfb3c04a33761b36346e3033bd88 not found: ID does not exist" Nov 03 09:17:08.924676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.784553 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3449b460-2c34-428a-a2c7-71b5dba3b16a-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:08.924676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:08.784583 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjmpt\" (UniqueName: \"kubernetes.io/projected/3449b460-2c34-428a-a2c7-71b5dba3b16a-kube-api-access-rjmpt\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:09.021597 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.007789 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-133b31-86f755d454-724sj"] Nov 03 09:17:09.442515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.440706 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:17:09.504509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.496186 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 09:17:09.512508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.505595 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02a39f0-92d4-4963-8be6-18dbe4da520e" path="/var/lib/kubelet/pods/f02a39f0-92d4-4963-8be6-18dbe4da520e/volumes" Nov 03 09:17:09.547501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.546614 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" podUID="3449b460-2c34-428a-a2c7-71b5dba3b16a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.153:50052\" within 1s: context deadline exceeded" Nov 03 09:17:09.547501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.546747 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" podUID="3f67346f-7c34-460d-898e-30dd07e22f12" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.152:50052\" within 1s: context deadline exceeded" Nov 03 09:17:09.547501 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:17:09.546867 5015 logging.go:55] [core] [Channel #4600 SubChannel #4601]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.153:50052", ServerName: "10.132.0.153:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.153:50052: operation was canceled" Nov 03 09:17:09.661572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.650397 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f67346f-7c34-460d-898e-30dd07e22f12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f67346f-7c34-460d-898e-30dd07e22f12-plugins-config-volume\") pod \"3f67346f-7c34-460d-898e-30dd07e22f12\" (UID: \"3f67346f-7c34-460d-898e-30dd07e22f12\") " Nov 03 09:17:09.661572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.650517 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f67346f-7c34-460d-898e-30dd07e22f12-kube-api-access-rsxx6\" (UniqueName: \"kubernetes.io/projected/3f67346f-7c34-460d-898e-30dd07e22f12-kube-api-access-rsxx6\") pod \"3f67346f-7c34-460d-898e-30dd07e22f12\" (UID: \"3f67346f-7c34-460d-898e-30dd07e22f12\") " Nov 03 09:17:09.739492 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.690714 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f67346f-7c34-460d-898e-30dd07e22f12-plugins-config-volume" pod "3f67346f-7c34-460d-898e-30dd07e22f12" (UID: "3f67346f-7c34-460d-898e-30dd07e22f12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:17:09.767576 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.767524 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f67346f-7c34-460d-898e-30dd07e22f12-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:09.796506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.792335 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" event={"ID":"3f67346f-7c34-460d-898e-30dd07e22f12","Type":"ContainerDied","Data":"228d02b36efd7dc3b21a1009cd63d681080b04254fc00ed3bbc4527dae11181a"} Nov 03 09:17:09.796506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.792414 5015 scope.go:122] "RemoveContainer" containerID="139c81dbe0a097974500d993503c57e89ac7cebd67b521c89122ff28c1040c4b" Nov 03 09:17:09.805804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.805772 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77" Nov 03 09:17:10.054563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.980046 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f67346f-7c34-460d-898e-30dd07e22f12-kube-api-access-rsxx6" pod "3f67346f-7c34-460d-898e-30dd07e22f12" (UID: "3f67346f-7c34-460d-898e-30dd07e22f12"). InnerVolumeSpecName "kube-api-access-rsxx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:10.054563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:09.985025 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsxx6\" (UniqueName: \"kubernetes.io/projected/3f67346f-7c34-460d-898e-30dd07e22f12-kube-api-access-rsxx6\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:10.181552 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.172546 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:17:10.295515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.288068 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" event={"ID":"3449b460-2c34-428a-a2c7-71b5dba3b16a","Type":"ContainerDied","Data":"fa93cd7b4cfc2e79d70b0f3603d2a45fb1adc817ff20494ef1e3929005f9ab26"} Nov 03 09:17:10.295515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.288148 5015 scope.go:122] "RemoveContainer" containerID="5dfbb7ceea0a66fd725f1fa82c31dfa3e2a2944d4cc6468bd26802472dc29065" Nov 03 09:17:10.295515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.288265 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7" Nov 03 09:17:10.327550 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.319878 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Nov 03 09:17:10.368725 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.364813 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:17:10.368725 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.365572 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 09:17:10.405521 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.395428 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 09:17:10.412935 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.412894 5015 generic.go:356] "Generic (PLEG): container finished" podID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerID="9810235e28f946dd422a893006a60cd65212c61893614fec96c8aca6151de1db" exitCode=0 Nov 03 09:17:10.413193 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.413170 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" event={"ID":"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b","Type":"ContainerDied","Data":"9810235e28f946dd422a893006a60cd65212c61893614fec96c8aca6151de1db"} Nov 03 09:17:10.453669 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.453625 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-certs\") pod \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " Nov 03 09:17:10.453880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.453709 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-podinfo\") pod \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " Nov 03 09:17:10.453880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.453785 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-envoy\") pod \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " Nov 03 09:17:10.453880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.453849 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-token\" (UniqueName: \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-token\") pod \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " Nov 03 09:17:10.454065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.453896 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-socket\") pod \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " Nov 03 09:17:10.454065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.453972 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-data\") pod \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " Nov 03 09:17:10.454189 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.454071 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-credential-socket\") pod \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " Nov 03 09:17:10.454189 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.454138 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-kube-api-access-khg2z\" (UniqueName: \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-kube-api-access-khg2z\") pod \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " Nov 03 09:17:10.454305 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.454211 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istiod-ca-cert\") pod \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\" (UID: \"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b\") " Nov 03 09:17:10.455127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.455094 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-data" pod "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" (UID: "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:10.455458 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.455437 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-socket" pod "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" (UID: "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:10.465624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.465592 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-credential-socket" pod "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" (UID: "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:10.466281 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.466249 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6" podUID="21ad0cfe-e3c6-4770-b0b3-7c512e7f1830" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:17:10.471505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.466755 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 09:17:10.475512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.474903 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-certs" pod "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" (UID: "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:10.526504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.517708 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-token" pod "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" (UID: "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:10.526504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.518262 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-envoy" pod "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" (UID: "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:10.526504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.518681 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-kube-api-access-khg2z" pod "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" (UID: "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b"). InnerVolumeSpecName "kube-api-access-khg2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:10.551503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.536252 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istiod-ca-cert" pod "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" (UID: "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:17:10.551503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.549545 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77"] Nov 03 09:17:10.551503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.549983 5015 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:17:10.597254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.597164 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-133b31-78d58c769f-65w77"] Nov 03 09:17:10.609849 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.598714 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khg2z\" (UniqueName: \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-kube-api-access-khg2z\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:10.610063 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.610047 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:10.610152 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.610141 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:10.610225 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.610218 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:10.610303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.610295 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:10.610378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.610370 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:10.610445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.610438 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:10.610525 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.610516 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:10.611153 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.611115 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-podinfo" pod "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" (UID: "fc56a0ca-ca23-4ef2-a13b-3368c3944e8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:17:10.681575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.678895 5015 scope.go:122] "RemoveContainer" containerID="9810235e28f946dd422a893006a60cd65212c61893614fec96c8aca6151de1db" Nov 03 09:17:10.713640 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.713564 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:10.738017 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.737630 5015 scope.go:122] "RemoveContainer" containerID="a84a08297e12eea2f078d0c1e2a2f8b206b6709c2d03c8db057130c1d05e00a4" Nov 03 09:17:10.795725 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.795692 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7"] Nov 03 09:17:10.801161 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.801135 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-133b31-58b6d6b75c-rrbj7"] Nov 03 09:17:10.919020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.918942 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:17:10.919020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:10.919007 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:17:11.206427 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.206341 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3449b460-2c34-428a-a2c7-71b5dba3b16a" path="/var/lib/kubelet/pods/3449b460-2c34-428a-a2c7-71b5dba3b16a/volumes" Nov 03 09:17:11.207298 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.207275 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f67346f-7c34-460d-898e-30dd07e22f12" path="/var/lib/kubelet/pods/3f67346f-7c34-460d-898e-30dd07e22f12/volumes" Nov 03 09:17:11.427263 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.427234 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" Nov 03 09:17:11.427442 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.427274 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw" event={"ID":"fc56a0ca-ca23-4ef2-a13b-3368c3944e8b","Type":"ContainerDied","Data":"5428c13964cde99a61e05003182a3192114baec7203f8af417eb57d068d2e561"} Nov 03 09:17:11.603129 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.603092 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw"] Nov 03 09:17:11.651420 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.651386 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-121ee6-openshift-default-6f8b75fcbf-kprbw"] Nov 03 09:17:11.842375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.842306 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:17:11.929440 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.929408 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-data\") pod \"e9c714c4-b970-4221-ab53-f6720684ba47\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " Nov 03 09:17:11.929619 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.929449 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-certs\") pod \"e9c714c4-b970-4221-ab53-f6720684ba47\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " Nov 03 09:17:11.929619 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.929521 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-istio-token\" (UniqueName: \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-istio-token\") pod \"e9c714c4-b970-4221-ab53-f6720684ba47\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " Nov 03 09:17:11.929619 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.929569 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c714c4-b970-4221-ab53-f6720684ba47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c714c4-b970-4221-ab53-f6720684ba47-istiod-ca-cert\") pod \"e9c714c4-b970-4221-ab53-f6720684ba47\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " Nov 03 09:17:11.929765 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.929748 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-kube-api-access-rlzgd\" (UniqueName: \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-kube-api-access-rlzgd\") pod \"e9c714c4-b970-4221-ab53-f6720684ba47\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " Nov 03 09:17:11.929819 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.929774 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-data" pod "e9c714c4-b970-4221-ab53-f6720684ba47" (UID: "e9c714c4-b970-4221-ab53-f6720684ba47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:11.929877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.929829 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-socket\") pod \"e9c714c4-b970-4221-ab53-f6720684ba47\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " Nov 03 09:17:11.929877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.929848 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-certs" pod "e9c714c4-b970-4221-ab53-f6720684ba47" (UID: "e9c714c4-b970-4221-ab53-f6720684ba47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:11.929877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.929864 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9c714c4-b970-4221-ab53-f6720684ba47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c714c4-b970-4221-ab53-f6720684ba47-istio-podinfo\") pod \"e9c714c4-b970-4221-ab53-f6720684ba47\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " Nov 03 09:17:11.930012 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.929955 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c714c4-b970-4221-ab53-f6720684ba47-istiod-ca-cert" pod "e9c714c4-b970-4221-ab53-f6720684ba47" (UID: "e9c714c4-b970-4221-ab53-f6720684ba47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:17:11.930056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.930035 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-envoy\") pod \"e9c714c4-b970-4221-ab53-f6720684ba47\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " Nov 03 09:17:11.930148 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.930125 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-socket" pod "e9c714c4-b970-4221-ab53-f6720684ba47" (UID: "e9c714c4-b970-4221-ab53-f6720684ba47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:11.930239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.930222 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-credential-socket\") pod \"e9c714c4-b970-4221-ab53-f6720684ba47\" (UID: \"e9c714c4-b970-4221-ab53-f6720684ba47\") " Nov 03 09:17:11.930733 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.930707 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-credential-socket" pod "e9c714c4-b970-4221-ab53-f6720684ba47" (UID: "e9c714c4-b970-4221-ab53-f6720684ba47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:11.930976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.930955 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:11.931020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.930985 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:11.931020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.931002 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:11.931079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.931017 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c714c4-b970-4221-ab53-f6720684ba47-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:11.931079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.931032 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:11.932062 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.932043 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9c714c4-b970-4221-ab53-f6720684ba47-istio-podinfo" pod "e9c714c4-b970-4221-ab53-f6720684ba47" (UID: "e9c714c4-b970-4221-ab53-f6720684ba47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:17:11.932211 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.932197 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-envoy" pod "e9c714c4-b970-4221-ab53-f6720684ba47" (UID: "e9c714c4-b970-4221-ab53-f6720684ba47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:11.932571 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.932545 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-kube-api-access-rlzgd" pod "e9c714c4-b970-4221-ab53-f6720684ba47" (UID: "e9c714c4-b970-4221-ab53-f6720684ba47"). InnerVolumeSpecName "kube-api-access-rlzgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:11.932659 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:11.932576 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-istio-token" pod "e9c714c4-b970-4221-ab53-f6720684ba47" (UID: "e9c714c4-b970-4221-ab53-f6720684ba47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:12.032302 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.032271 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlzgd\" (UniqueName: \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-kube-api-access-rlzgd\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:12.032302 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.032300 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c714c4-b970-4221-ab53-f6720684ba47-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:12.032536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.032312 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c714c4-b970-4221-ab53-f6720684ba47-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:12.032536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.032321 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c714c4-b970-4221-ab53-f6720684ba47-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:12.439257 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.439219 5015 generic.go:356] "Generic (PLEG): container finished" podID="e9c714c4-b970-4221-ab53-f6720684ba47" containerID="a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966" exitCode=0 Nov 03 09:17:12.439447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.439274 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" event={"ID":"e9c714c4-b970-4221-ab53-f6720684ba47","Type":"ContainerDied","Data":"a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966"} Nov 03 09:17:12.439447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.439290 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" Nov 03 09:17:12.439447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.439349 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84" event={"ID":"e9c714c4-b970-4221-ab53-f6720684ba47","Type":"ContainerDied","Data":"2f304ed9f55f197bf1eddcaef48be8e3cccd47477a8380c27459a2ad457504fc"} Nov 03 09:17:12.439447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.439381 5015 scope.go:122] "RemoveContainer" containerID="a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966" Nov 03 09:17:12.454404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.454386 5015 scope.go:122] "RemoveContainer" containerID="a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966" Nov 03 09:17:12.454759 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:12.454740 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966\": container with ID starting with a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966 not found: ID does not exist" containerID="a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966" Nov 03 09:17:12.454827 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.454771 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966"} err="failed to get container status \"a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966\": rpc error: code = NotFound desc = could not find container \"a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966\": container with ID starting with a59ea5bc631666f41cdc3c67c762edffd107d45e474f1b6d765327c22a723966 not found: ID does not exist" Nov 03 09:17:12.613518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.613464 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84"] Nov 03 09:17:12.664097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:12.664056 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-hmx84"] Nov 03 09:17:13.204875 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:13.204843 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" path="/var/lib/kubelet/pods/e9c714c4-b970-4221-ab53-f6720684ba47/volumes" Nov 03 09:17:13.205319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:13.205304 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" path="/var/lib/kubelet/pods/fc56a0ca-ca23-4ef2-a13b-3368c3944e8b/volumes" Nov 03 09:17:16.003321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.003270 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r"] Nov 03 09:17:16.003959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.003941 5015 cpu_manager.go:414] "containerMap: removing container" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" containerName="istio-proxy" Nov 03 09:17:16.004008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.003961 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" containerName="istio-proxy" Nov 03 09:17:16.004008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.003985 5015 cpu_manager.go:414] "containerMap: removing container" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" Nov 03 09:17:16.004008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.003990 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" Nov 03 09:17:16.004008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004001 5015 cpu_manager.go:414] "containerMap: removing container" podUID="3449b460-2c34-428a-a2c7-71b5dba3b16a" containerName="payload-processing" Nov 03 09:17:16.004008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004006 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3449b460-2c34-428a-a2c7-71b5dba3b16a" containerName="payload-processing" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004013 5015 cpu_manager.go:414] "containerMap: removing container" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004017 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004024 5015 cpu_manager.go:414] "containerMap: removing container" podUID="f02a39f0-92d4-4963-8be6-18dbe4da520e" containerName="maas-api" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004029 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02a39f0-92d4-4963-8be6-18dbe4da520e" containerName="maas-api" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004043 5015 cpu_manager.go:414] "containerMap: removing container" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004047 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004054 5015 cpu_manager.go:414] "containerMap: removing container" podUID="3f67346f-7c34-460d-898e-30dd07e22f12" containerName="payload-pre-processing" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004058 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f67346f-7c34-460d-898e-30dd07e22f12" containerName="payload-pre-processing" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004066 5015 cpu_manager.go:414] "containerMap: removing container" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004070 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004077 5015 cpu_manager.go:414] "containerMap: removing container" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" Nov 03 09:17:16.004160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004081 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" Nov 03 09:17:16.004542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004174 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c714c4-b970-4221-ab53-f6720684ba47" containerName="istio-proxy" Nov 03 09:17:16.004542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004185 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6cff466-5838-4ed8-9124-09bdc768a0ca" containerName="istio-proxy" Nov 03 09:17:16.004542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004193 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="3449b460-2c34-428a-a2c7-71b5dba3b16a" containerName="payload-processing" Nov 03 09:17:16.004542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004199 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" Nov 03 09:17:16.004542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004205 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02a39f0-92d4-4963-8be6-18dbe4da520e" containerName="maas-api" Nov 03 09:17:16.004542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004213 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" Nov 03 09:17:16.004542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004217 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f67346f-7c34-460d-898e-30dd07e22f12" containerName="payload-pre-processing" Nov 03 09:17:16.004542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004223 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f961ac-69cf-407a-af9b-07611b20726c" containerName="istio-proxy" Nov 03 09:17:16.004542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.004229 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc56a0ca-ca23-4ef2-a13b-3368c3944e8b" containerName="istio-proxy" Nov 03 09:17:16.008742 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.008727 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.063866 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.063829 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r"] Nov 03 09:17:16.084576 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.084549 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fplm\" (UniqueName: \"kubernetes.io/projected/10994f28-5c15-4e2d-b97a-40680160270d-kube-api-access-7fplm\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.084728 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.084621 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10994f28-5c15-4e2d-b97a-40680160270d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.084728 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.084681 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-metrics-certs\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.084930 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.084909 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.110157 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.110135 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-1ce7a7-serving-cert\"" Nov 03 09:17:16.185658 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.185624 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10994f28-5c15-4e2d-b97a-40680160270d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.185658 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.185657 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-metrics-certs\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.186020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.185773 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.186020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.185816 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fplm\" (UniqueName: \"kubernetes.io/projected/10994f28-5c15-4e2d-b97a-40680160270d-kube-api-access-7fplm\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.186020 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:16.185902 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-serving-cert: secret "maas-api-e2e-praxis-a-1ce7a7-serving-cert" not found Nov 03 09:17:16.186020 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:16.185988 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-maas-api-tls podName:10994f28-5c15-4e2d-b97a-40680160270d nodeName:}" failed. No retries permitted until 2026-11-03 09:17:16.685972483 +0000 UTC m=+2269.610914681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-maas-api-tls") pod "maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" (UID: "10994f28-5c15-4e2d-b97a-40680160270d") : secret "maas-api-e2e-praxis-a-1ce7a7-serving-cert" not found Nov 03 09:17:16.186326 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.186256 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10994f28-5c15-4e2d-b97a-40680160270d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.188293 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.188269 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-metrics-certs\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.246625 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.246596 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fplm\" (UniqueName: \"kubernetes.io/projected/10994f28-5c15-4e2d-b97a-40680160270d-kube-api-access-7fplm\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.693102 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.693065 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.695774 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.695753 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:16.902293 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:16.902251 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:17:16.925110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:16.925082 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:17.313194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:17.313166 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r"] Nov 03 09:17:17.314644 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:17:17.314618 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10994f28_5c15_4e2d_b97a_40680160270d.slice/crio-9e65a92a3158061a0e5a31e935b2625ee24492907d87730a22f83b6c422211c2 WatchSource:0}: Error finding container 9e65a92a3158061a0e5a31e935b2625ee24492907d87730a22f83b6c422211c2: Status 404 returned error can't find the container with id 9e65a92a3158061a0e5a31e935b2625ee24492907d87730a22f83b6c422211c2 Nov 03 09:17:17.316730 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:17.316707 5015 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:17:17.487118 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:17.487085 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" event={"ID":"10994f28-5c15-4e2d-b97a-40680160270d","Type":"ContainerStarted","Data":"9e65a92a3158061a0e5a31e935b2625ee24492907d87730a22f83b6c422211c2"} Nov 03 09:17:18.113434 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:18.113404 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq"] Nov 03 09:17:18.113696 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:18.113669 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" containerID="cri-o://b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75" gracePeriod=30 Nov 03 09:17:18.495140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:18.495103 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" event={"ID":"10994f28-5c15-4e2d-b97a-40680160270d","Type":"ContainerStarted","Data":"ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c"} Nov 03 09:17:18.495609 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:18.495335 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:18.615198 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:18.615148 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" podStartSLOduration=2.723213198 podStartE2EDuration="3.615134435s" totalImagesPullingTime="891.921237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:17:18.614675041 +0000 UTC m=+2271.539617260" watchObservedRunningTime="2026-11-03 09:17:18.615134435 +0000 UTC m=+2271.540076652" Nov 03 09:17:23.387561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.387493 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:17:23.479909 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.479879 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-envoy\") pod \"2510f27d-c29c-43b0-bf06-43dc2d174f56\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " Nov 03 09:17:23.480098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.479921 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-data\") pod \"2510f27d-c29c-43b0-bf06-43dc2d174f56\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " Nov 03 09:17:23.480098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.479953 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-credential-socket\") pod \"2510f27d-c29c-43b0-bf06-43dc2d174f56\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " Nov 03 09:17:23.480098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.480016 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2510f27d-c29c-43b0-bf06-43dc2d174f56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2510f27d-c29c-43b0-bf06-43dc2d174f56-istiod-ca-cert\") pod \"2510f27d-c29c-43b0-bf06-43dc2d174f56\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " Nov 03 09:17:23.480098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.480046 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-kube-api-access-z9rlq\" (UniqueName: \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-kube-api-access-z9rlq\") pod \"2510f27d-c29c-43b0-bf06-43dc2d174f56\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " Nov 03 09:17:23.480337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.480103 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-token\" (UniqueName: \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-token\") pod \"2510f27d-c29c-43b0-bf06-43dc2d174f56\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " Nov 03 09:17:23.480337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.480138 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-podinfo\") pod \"2510f27d-c29c-43b0-bf06-43dc2d174f56\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " Nov 03 09:17:23.480337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.480195 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-data" pod "2510f27d-c29c-43b0-bf06-43dc2d174f56" (UID: "2510f27d-c29c-43b0-bf06-43dc2d174f56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:23.480337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.480258 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-credential-socket" pod "2510f27d-c29c-43b0-bf06-43dc2d174f56" (UID: "2510f27d-c29c-43b0-bf06-43dc2d174f56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:23.480337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.480265 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-socket\") pod \"2510f27d-c29c-43b0-bf06-43dc2d174f56\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " Nov 03 09:17:23.480709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.480361 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-certs\") pod \"2510f27d-c29c-43b0-bf06-43dc2d174f56\" (UID: \"2510f27d-c29c-43b0-bf06-43dc2d174f56\") " Nov 03 09:17:23.480709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.480510 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-socket" pod "2510f27d-c29c-43b0-bf06-43dc2d174f56" (UID: "2510f27d-c29c-43b0-bf06-43dc2d174f56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:23.480830 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.480713 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2510f27d-c29c-43b0-bf06-43dc2d174f56-istiod-ca-cert" pod "2510f27d-c29c-43b0-bf06-43dc2d174f56" (UID: "2510f27d-c29c-43b0-bf06-43dc2d174f56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:17:23.480897 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.480840 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-certs" pod "2510f27d-c29c-43b0-bf06-43dc2d174f56" (UID: "2510f27d-c29c-43b0-bf06-43dc2d174f56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:23.482101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.481099 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:23.482101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.481121 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:23.482101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.481132 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:23.482101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.481144 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:23.482101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.481191 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2510f27d-c29c-43b0-bf06-43dc2d174f56-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:23.482926 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.482901 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-envoy" pod "2510f27d-c29c-43b0-bf06-43dc2d174f56" (UID: "2510f27d-c29c-43b0-bf06-43dc2d174f56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:23.483044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.483013 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-kube-api-access-z9rlq" pod "2510f27d-c29c-43b0-bf06-43dc2d174f56" (UID: "2510f27d-c29c-43b0-bf06-43dc2d174f56"). InnerVolumeSpecName "kube-api-access-z9rlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:23.483128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.483064 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-token" pod "2510f27d-c29c-43b0-bf06-43dc2d174f56" (UID: "2510f27d-c29c-43b0-bf06-43dc2d174f56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:23.483248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.483228 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-podinfo" pod "2510f27d-c29c-43b0-bf06-43dc2d174f56" (UID: "2510f27d-c29c-43b0-bf06-43dc2d174f56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:17:23.530610 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.530579 5015 generic.go:356] "Generic (PLEG): container finished" podID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerID="b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75" exitCode=0 Nov 03 09:17:23.530742 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.530641 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" Nov 03 09:17:23.530742 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.530667 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" event={"ID":"2510f27d-c29c-43b0-bf06-43dc2d174f56","Type":"ContainerDied","Data":"b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75"} Nov 03 09:17:23.530742 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.530726 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq" event={"ID":"2510f27d-c29c-43b0-bf06-43dc2d174f56","Type":"ContainerDied","Data":"8396abb0bc234bb09a2becfcc37d537b2bdfe4bf37e3fb22c6c412a03be2e00a"} Nov 03 09:17:23.530909 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.530749 5015 scope.go:122] "RemoveContainer" containerID="b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75" Nov 03 09:17:23.542298 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.542277 5015 scope.go:122] "RemoveContainer" containerID="17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555" Nov 03 09:17:23.551516 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.551499 5015 scope.go:122] "RemoveContainer" containerID="b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75" Nov 03 09:17:23.551770 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:23.551755 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75\": container with ID starting with b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75 not found: ID does not exist" containerID="b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75" Nov 03 09:17:23.551810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.551779 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75"} err="failed to get container status \"b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75\": rpc error: code = NotFound desc = could not find container \"b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75\": container with ID starting with b65e491a6c580272ad1efdaddbe6118d45c74399e7d4ae4e368c2c78e4603d75 not found: ID does not exist" Nov 03 09:17:23.551810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.551796 5015 scope.go:122] "RemoveContainer" containerID="17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555" Nov 03 09:17:23.552024 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:23.552007 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555\": container with ID starting with 17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555 not found: ID does not exist" containerID="17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555" Nov 03 09:17:23.552061 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.552032 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555"} err="failed to get container status \"17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555\": rpc error: code = NotFound desc = could not find container \"17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555\": container with ID starting with 17699cdf79c22da98be78c42c076b25f41c890a35770aad609f112897edf3555 not found: ID does not exist" Nov 03 09:17:23.582041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.582016 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:23.582041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.582040 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9rlq\" (UniqueName: \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-kube-api-access-z9rlq\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:23.582202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.582050 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:23.582202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.582059 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2510f27d-c29c-43b0-bf06-43dc2d174f56-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:23.676169 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.676093 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq"] Nov 03 09:17:23.679139 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:23.679117 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-133b31-openshift-default-84fb5945b5-8z9sq"] Nov 03 09:17:24.508000 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:24.507970 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:17:24.679496 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:24.679254 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v"] Nov 03 09:17:25.205896 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:25.205858 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" path="/var/lib/kubelet/pods/2510f27d-c29c-43b0-bf06-43dc2d174f56/volumes" Nov 03 09:17:29.680223 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:29.680168 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" podUID="2b46ba45-e329-44cd-b50a-cddafe821ba8" containerName="payload-processing" containerID="cri-o://648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81" gracePeriod=25 Nov 03 09:17:29.970329 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:29.970299 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:17:29.981667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:29.981641 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b46ba45-e329-44cd-b50a-cddafe821ba8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b46ba45-e329-44cd-b50a-cddafe821ba8-plugins-config-volume\") pod \"2b46ba45-e329-44cd-b50a-cddafe821ba8\" (UID: \"2b46ba45-e329-44cd-b50a-cddafe821ba8\") " Nov 03 09:17:29.981795 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:29.981758 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b46ba45-e329-44cd-b50a-cddafe821ba8-kube-api-access-8lrs9\" (UniqueName: \"kubernetes.io/projected/2b46ba45-e329-44cd-b50a-cddafe821ba8-kube-api-access-8lrs9\") pod \"2b46ba45-e329-44cd-b50a-cddafe821ba8\" (UID: \"2b46ba45-e329-44cd-b50a-cddafe821ba8\") " Nov 03 09:17:29.982065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:29.982036 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b46ba45-e329-44cd-b50a-cddafe821ba8-plugins-config-volume" pod "2b46ba45-e329-44cd-b50a-cddafe821ba8" (UID: "2b46ba45-e329-44cd-b50a-cddafe821ba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:17:29.984028 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:29.984006 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b46ba45-e329-44cd-b50a-cddafe821ba8-kube-api-access-8lrs9" pod "2b46ba45-e329-44cd-b50a-cddafe821ba8" (UID: "2b46ba45-e329-44cd-b50a-cddafe821ba8"). InnerVolumeSpecName "kube-api-access-8lrs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:30.082845 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:30.082815 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b46ba45-e329-44cd-b50a-cddafe821ba8-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:30.082845 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:30.082843 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lrs9\" (UniqueName: \"kubernetes.io/projected/2b46ba45-e329-44cd-b50a-cddafe821ba8-kube-api-access-8lrs9\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:30.591709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:30.591672 5015 generic.go:356] "Generic (PLEG): container finished" podID="2b46ba45-e329-44cd-b50a-cddafe821ba8" containerID="648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81" exitCode=0 Nov 03 09:17:30.591901 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:30.591726 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" Nov 03 09:17:30.591901 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:30.591726 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" event={"ID":"2b46ba45-e329-44cd-b50a-cddafe821ba8","Type":"ContainerDied","Data":"648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81"} Nov 03 09:17:30.591901 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:30.591846 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v" event={"ID":"2b46ba45-e329-44cd-b50a-cddafe821ba8","Type":"ContainerDied","Data":"81406b754431a3b22be8c0101848a3f55205bf33ccb2a01eaf9b54757d8d0ea9"} Nov 03 09:17:30.591901 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:30.591877 5015 scope.go:122] "RemoveContainer" containerID="648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81" Nov 03 09:17:30.603167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:30.603141 5015 scope.go:122] "RemoveContainer" containerID="648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81" Nov 03 09:17:30.603538 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:30.603502 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81\": container with ID starting with 648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81 not found: ID does not exist" containerID="648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81" Nov 03 09:17:30.603644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:30.603549 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81"} err="failed to get container status \"648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81\": rpc error: code = NotFound desc = could not find container \"648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81\": container with ID starting with 648fdfaaac7e4af91ba3a2b8e4139b2ee38670e6d26013b47114bf723e7b2a81 not found: ID does not exist" Nov 03 09:17:30.765973 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:30.765935 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v"] Nov 03 09:17:30.769813 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:30.769788 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0eecfd-6dc497f779-7q88v"] Nov 03 09:17:31.204922 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:31.204883 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b46ba45-e329-44cd-b50a-cddafe821ba8" path="/var/lib/kubelet/pods/2b46ba45-e329-44cd-b50a-cddafe821ba8/volumes" Nov 03 09:17:36.308224 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:36.307990 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 09:17:42.112864 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:42.112830 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn"] Nov 03 09:17:42.113373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:42.113080 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" containerID="cri-o://d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599" gracePeriod=30 Nov 03 09:17:46.904544 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:46.904504 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:17:47.146536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.146509 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh"] Nov 03 09:17:47.147115 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.147098 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" Nov 03 09:17:47.147178 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.147117 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" Nov 03 09:17:47.147178 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.147138 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" Nov 03 09:17:47.147178 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.147143 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" Nov 03 09:17:47.147178 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.147166 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2b46ba45-e329-44cd-b50a-cddafe821ba8" containerName="payload-processing" Nov 03 09:17:47.147178 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.147170 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b46ba45-e329-44cd-b50a-cddafe821ba8" containerName="payload-processing" Nov 03 09:17:47.147333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.147256 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b46ba45-e329-44cd-b50a-cddafe821ba8" containerName="payload-processing" Nov 03 09:17:47.147333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.147268 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" Nov 03 09:17:47.150703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.150686 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.232790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.232761 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-433821-openshift-default-dockercfg-f7qkc\"" Nov 03 09:17:47.269526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.267235 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-envoy\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.269526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.267278 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-data\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.269526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.267304 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istiod-ca-cert\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.269526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.267324 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rgt\" (UniqueName: \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-kube-api-access-w9rgt\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.269526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.267369 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-certs\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.269526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.267400 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-podinfo\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.269526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.267419 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-credential-socket\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.269526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.267470 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-token\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.269526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.267512 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-socket\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.287067 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.287033 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh"] Nov 03 09:17:47.375539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.371943 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-envoy\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.375539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.371991 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-data\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.375539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.372040 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istiod-ca-cert\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.375539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.372088 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rgt\" (UniqueName: \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-kube-api-access-w9rgt\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.375539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.372168 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-certs\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.375539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.372222 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-podinfo\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.375539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.372248 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-credential-socket\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.375539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.372339 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-token\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.375539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.372373 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-socket\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.375539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.373307 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-certs\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.378587 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.378539 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-data\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.379402 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.379379 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-socket\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.379567 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.379545 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-envoy\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.379644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.379576 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-credential-socket\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.379852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.379833 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-podinfo\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.380152 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.380134 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istiod-ca-cert\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.419818 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.419795 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:17:47.432702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.432673 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-token\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.432907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.432886 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rgt\" (UniqueName: \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-kube-api-access-w9rgt\") pod \"e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.547709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.547619 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.575248 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-token\" (UniqueName: \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-token\") pod \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.575325 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-kube-api-access-4j77q\" (UniqueName: \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-kube-api-access-4j77q\") pod \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.575456 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-podinfo\") pod \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.575497 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-data\") pod \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.575565 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-socket\") pod \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.575918 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-data" pod "37ebc178-9bdf-4b0d-9d34-13b9fc475477" (UID: "37ebc178-9bdf-4b0d-9d34-13b9fc475477"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.576104 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-socket" pod "37ebc178-9bdf-4b0d-9d34-13b9fc475477" (UID: "37ebc178-9bdf-4b0d-9d34-13b9fc475477"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.576295 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-envoy\") pod \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.577567 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istiod-ca-cert\") pod \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.577637 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-certs\") pod \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.577675 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-credential-socket\") pod \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\" (UID: \"37ebc178-9bdf-4b0d-9d34-13b9fc475477\") " Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.577974 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-certs" pod "37ebc178-9bdf-4b0d-9d34-13b9fc475477" (UID: "37ebc178-9bdf-4b0d-9d34-13b9fc475477"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.578351 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istiod-ca-cert" pod "37ebc178-9bdf-4b0d-9d34-13b9fc475477" (UID: "37ebc178-9bdf-4b0d-9d34-13b9fc475477"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.578988 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.579106 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.579121 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.579142 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:47.584691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.579584 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-credential-socket" pod "37ebc178-9bdf-4b0d-9d34-13b9fc475477" (UID: "37ebc178-9bdf-4b0d-9d34-13b9fc475477"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:47.588348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.588318 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-podinfo" pod "37ebc178-9bdf-4b0d-9d34-13b9fc475477" (UID: "37ebc178-9bdf-4b0d-9d34-13b9fc475477"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:17:47.588542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.588519 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-token" pod "37ebc178-9bdf-4b0d-9d34-13b9fc475477" (UID: "37ebc178-9bdf-4b0d-9d34-13b9fc475477"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:47.588664 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.588642 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-envoy" pod "37ebc178-9bdf-4b0d-9d34-13b9fc475477" (UID: "37ebc178-9bdf-4b0d-9d34-13b9fc475477"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:17:47.589956 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.589927 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-kube-api-access-4j77q" pod "37ebc178-9bdf-4b0d-9d34-13b9fc475477" (UID: "37ebc178-9bdf-4b0d-9d34-13b9fc475477"). InnerVolumeSpecName "kube-api-access-4j77q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:47.681383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.681280 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j77q\" (UniqueName: \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-kube-api-access-4j77q\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:47.681383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.681319 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:47.681383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.681334 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:47.681383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.681347 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ebc178-9bdf-4b0d-9d34-13b9fc475477-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:47.681383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.681359 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ebc178-9bdf-4b0d-9d34-13b9fc475477-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:47.821862 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.821770 5015 generic.go:356] "Generic (PLEG): container finished" podID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerID="d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599" exitCode=0 Nov 03 09:17:47.821862 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.821824 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" Nov 03 09:17:47.822082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.821858 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" event={"ID":"37ebc178-9bdf-4b0d-9d34-13b9fc475477","Type":"ContainerDied","Data":"d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599"} Nov 03 09:17:47.822082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.821917 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn" event={"ID":"37ebc178-9bdf-4b0d-9d34-13b9fc475477","Type":"ContainerDied","Data":"6617fb2ed312cc8dc613f1f9cb9b7da7ffa75ca915574d14f3bb782d2cb7533b"} Nov 03 09:17:47.822082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.821950 5015 scope.go:122] "RemoveContainer" containerID="d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599" Nov 03 09:17:47.832951 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.832927 5015 scope.go:122] "RemoveContainer" containerID="601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f" Nov 03 09:17:47.842904 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.842880 5015 scope.go:122] "RemoveContainer" containerID="d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599" Nov 03 09:17:47.843166 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:47.843148 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599\": container with ID starting with d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599 not found: ID does not exist" containerID="d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599" Nov 03 09:17:47.843229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.843179 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599"} err="failed to get container status \"d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599\": rpc error: code = NotFound desc = could not find container \"d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599\": container with ID starting with d4a5bfaae339b34cc6651d9683e71eff537b2c2997bb146fe1139086ed7cb599 not found: ID does not exist" Nov 03 09:17:47.843229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.843199 5015 scope.go:122] "RemoveContainer" containerID="601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f" Nov 03 09:17:47.843433 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:17:47.843417 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f\": container with ID starting with 601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f not found: ID does not exist" containerID="601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f" Nov 03 09:17:47.843550 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.843438 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f"} err="failed to get container status \"601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f\": rpc error: code = NotFound desc = could not find container \"601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f\": container with ID starting with 601c0e5402fcab3d73ce70e2cdc0fe453fb8e3a61e7f8a99874486ca342dd95f not found: ID does not exist" Nov 03 09:17:47.996079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:47.996028 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn"] Nov 03 09:17:48.000774 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:48.000741 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-0eecfd-openshift-default-585fbb79fc-8chcn"] Nov 03 09:17:48.048549 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:48.048520 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh"] Nov 03 09:17:48.049963 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:48.049917 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:17:48.050123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:48.050056 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:17:48.050198 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:48.050149 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:17:48.831301 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:48.831261 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" event={"ID":"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d","Type":"ContainerStarted","Data":"ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6"} Nov 03 09:17:48.831508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:48.831326 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" event={"ID":"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d","Type":"ContainerStarted","Data":"75a437dfc9d08c98eab44502a1e020921cfe8ccf3e96abb6df17c583620f9887"} Nov 03 09:17:48.963599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:48.963540 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" podStartSLOduration=1.963527129 podStartE2EDuration="1.963527129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:17:48.961787744 +0000 UTC m=+2301.886729970" watchObservedRunningTime="2026-11-03 09:17:48.963527129 +0000 UTC m=+2301.888469345" Nov 03 09:17:49.204998 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:49.204914 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" path="/var/lib/kubelet/pods/37ebc178-9bdf-4b0d-9d34-13b9fc475477/volumes" Nov 03 09:17:49.548468 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:49.548433 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:49.555156 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:49.555130 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:49.839675 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:49.839587 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:49.840665 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:49.840645 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:17:54.362327 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.362281 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g"] Nov 03 09:17:54.362859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.362843 5015 cpu_manager.go:414] "containerMap: removing container" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" Nov 03 09:17:54.362859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.362858 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" Nov 03 09:17:54.362984 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.362964 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" Nov 03 09:17:54.362984 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.362977 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" Nov 03 09:17:54.363084 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.362988 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2510f27d-c29c-43b0-bf06-43dc2d174f56" containerName="istio-proxy" Nov 03 09:17:54.366673 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.366653 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.472083 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.472049 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-910d50-openshift-default-dockercfg-w8xwd\"" Nov 03 09:17:54.472977 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.472946 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.473139 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.473019 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.473214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.473147 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs652\" (UniqueName: \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-kube-api-access-zs652\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.473214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.473191 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.473333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.473228 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.473333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.473258 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.473446 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.473332 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.473446 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.473362 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.473446 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.473390 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.476064 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.476035 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g"] Nov 03 09:17:54.574772 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.574734 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575017 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.574787 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575017 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.574824 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs652\" (UniqueName: \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-kube-api-access-zs652\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575017 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.574846 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575017 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.574867 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575017 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.574980 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575263 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.575073 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575263 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.575108 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575263 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.575144 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575466 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.575436 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575763 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.575739 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575870 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.575788 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.575870 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.575851 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.576204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.576187 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.577985 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.577965 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.578087 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.578023 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.634656 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.634587 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.635051 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.635029 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs652\" (UniqueName: \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-kube-api-access-zs652\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:54.680686 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:54.680651 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:55.127796 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.127765 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g"] Nov 03 09:17:55.152975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.152711 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:17:55.152975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.152802 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:17:55.152975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.152847 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:17:55.420379 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.420306 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f"] Nov 03 09:17:55.420923 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.420907 5015 cpu_manager.go:414] "containerMap: removing container" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" Nov 03 09:17:55.420976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.420925 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ebc178-9bdf-4b0d-9d34-13b9fc475477" containerName="istio-proxy" Nov 03 09:17:55.424543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.424527 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:17:55.482250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.482188 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-433821-dockercfg-bmgkg\"" Nov 03 09:17:55.482830 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.482805 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-433821\"" Nov 03 09:17:55.485931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.485904 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f"] Nov 03 09:17:55.491423 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.491396 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353fe68-ba7d-433c-b7b6-9dca79d7de10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f\" (UID: \"d353fe68-ba7d-433c-b7b6-9dca79d7de10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:17:55.491631 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.491611 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qc9c\" (UniqueName: \"kubernetes.io/projected/d353fe68-ba7d-433c-b7b6-9dca79d7de10-kube-api-access-2qc9c\") pod \"payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f\" (UID: \"d353fe68-ba7d-433c-b7b6-9dca79d7de10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:17:55.593565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.592970 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qc9c\" (UniqueName: \"kubernetes.io/projected/d353fe68-ba7d-433c-b7b6-9dca79d7de10-kube-api-access-2qc9c\") pod \"payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f\" (UID: \"d353fe68-ba7d-433c-b7b6-9dca79d7de10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:17:55.593565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.593146 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353fe68-ba7d-433c-b7b6-9dca79d7de10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f\" (UID: \"d353fe68-ba7d-433c-b7b6-9dca79d7de10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:17:55.606504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.594213 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353fe68-ba7d-433c-b7b6-9dca79d7de10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f\" (UID: \"d353fe68-ba7d-433c-b7b6-9dca79d7de10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:17:55.613352 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.610736 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4"] Nov 03 09:17:55.617326 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.617049 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:17:55.646182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.645291 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4"] Nov 03 09:17:55.663533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.658583 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qc9c\" (UniqueName: \"kubernetes.io/projected/d353fe68-ba7d-433c-b7b6-9dca79d7de10-kube-api-access-2qc9c\") pod \"payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f\" (UID: \"d353fe68-ba7d-433c-b7b6-9dca79d7de10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:17:55.694684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.694589 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shpwr\" (UniqueName: \"kubernetes.io/projected/4e5255ed-f323-4c2f-b431-14b4bcf956e4-kube-api-access-shpwr\") pod \"payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4\" (UID: \"4e5255ed-f323-4c2f-b431-14b4bcf956e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:17:55.694842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.694753 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5255ed-f323-4c2f-b431-14b4bcf956e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4\" (UID: \"4e5255ed-f323-4c2f-b431-14b4bcf956e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:17:55.741513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.741437 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:17:55.795986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.795953 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shpwr\" (UniqueName: \"kubernetes.io/projected/4e5255ed-f323-4c2f-b431-14b4bcf956e4-kube-api-access-shpwr\") pod \"payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4\" (UID: \"4e5255ed-f323-4c2f-b431-14b4bcf956e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:17:55.796169 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.796067 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5255ed-f323-4c2f-b431-14b4bcf956e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4\" (UID: \"4e5255ed-f323-4c2f-b431-14b4bcf956e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:17:55.797044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.797020 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5255ed-f323-4c2f-b431-14b4bcf956e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4\" (UID: \"4e5255ed-f323-4c2f-b431-14b4bcf956e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:17:55.857530 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.855684 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shpwr\" (UniqueName: \"kubernetes.io/projected/4e5255ed-f323-4c2f-b431-14b4bcf956e4-kube-api-access-shpwr\") pod \"payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4\" (UID: \"4e5255ed-f323-4c2f-b431-14b4bcf956e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:17:55.886158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.886114 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" event={"ID":"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e","Type":"ContainerStarted","Data":"ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112"} Nov 03 09:17:55.886343 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.886186 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" event={"ID":"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e","Type":"ContainerStarted","Data":"837f147eb60c1c0402df0d41cbd8fa3b1c68f3c0f5e60a910fa039990f18edaf"} Nov 03 09:17:55.976502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:55.974823 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:17:56.009395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.009336 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" podStartSLOduration=2.009318506 podStartE2EDuration="2.009318506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:17:56.008385559 +0000 UTC m=+2308.933327777" watchObservedRunningTime="2026-11-03 09:17:56.009318506 +0000 UTC m=+2308.934260725" Nov 03 09:17:56.130357 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.130328 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f"] Nov 03 09:17:56.384096 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.384067 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4"] Nov 03 09:17:56.681945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.681849 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:17:56.683578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.683551 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 09:17:56.683713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.683604 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 09:17:56.894994 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.894898 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" event={"ID":"4e5255ed-f323-4c2f-b431-14b4bcf956e4","Type":"ContainerStarted","Data":"5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca"} Nov 03 09:17:56.894994 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.894964 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" event={"ID":"4e5255ed-f323-4c2f-b431-14b4bcf956e4","Type":"ContainerStarted","Data":"dad44f8e7000081d70aebac9db4f198d88b7f69634bf003ac2a638d6995e72f0"} Nov 03 09:17:56.895230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.895096 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:17:56.896691 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.896661 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" event={"ID":"d353fe68-ba7d-433c-b7b6-9dca79d7de10","Type":"ContainerStarted","Data":"d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f"} Nov 03 09:17:56.896822 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.896699 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" event={"ID":"d353fe68-ba7d-433c-b7b6-9dca79d7de10","Type":"ContainerStarted","Data":"d1ab493ea8c1fefd288e422279665f2b3f4366c8d5b62a402388b8c217fb9ea4"} Nov 03 09:17:56.896917 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:56.896895 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:17:57.018173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:57.018122 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" podStartSLOduration=2.018108115 podStartE2EDuration="2.018108115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:17:57.015630958 +0000 UTC m=+2309.940573194" watchObservedRunningTime="2026-11-03 09:17:57.018108115 +0000 UTC m=+2309.943050331" Nov 03 09:17:57.134074 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:57.134013 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" podStartSLOduration=2.133995684 podStartE2EDuration="2.133995684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:17:57.131930705 +0000 UTC m=+2310.056872924" watchObservedRunningTime="2026-11-03 09:17:57.133995684 +0000 UTC m=+2310.058937902" Nov 03 09:17:57.681765 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:57.681728 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 09:17:57.681975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:57.681800 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 09:17:58.682076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:58.682036 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 09:17:58.682498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:58.682104 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 09:17:59.681219 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:59.681177 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 09:17:59.681407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:17:59.681270 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 09:18:00.193198 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.193162 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6"] Nov 03 09:18:00.206110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.206073 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.278400 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.278360 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-certs\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.325515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.316104 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-data\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.341539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.335719 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-envoy\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.341539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.335848 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hk5h\" (UniqueName: \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-kube-api-access-7hk5h\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.341539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.335894 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-credential-socket\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.341539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.336154 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-socket\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.341539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.336241 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-token\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.341539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.336335 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-podinfo\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.341539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.336385 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a8c80d-0637-4e7f-81b9-e17e2523d291-istiod-ca-cert\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.341539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.279509 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-627e5f-openshift-default-dockercfg-bmpf2\"" Nov 03 09:18:00.429398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.429352 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6"] Nov 03 09:18:00.463545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.443265 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-podinfo\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.463860 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.463830 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a8c80d-0637-4e7f-81b9-e17e2523d291-istiod-ca-cert\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.464044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.464028 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-certs\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.464268 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.464251 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-data\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.472679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.472647 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-envoy\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.472923 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.472887 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hk5h\" (UniqueName: \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-kube-api-access-7hk5h\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.473085 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.473066 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-credential-socket\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.473401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.473382 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-socket\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.473578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.473566 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-token\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.479782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.477433 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-certs\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.479782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.477737 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-credential-socket\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.479782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.478224 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-socket\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.479782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.478508 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-data\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.479782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.478732 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-envoy\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.479782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.479548 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a8c80d-0637-4e7f-81b9-e17e2523d291-istiod-ca-cert\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.483180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.483156 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-podinfo\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.534128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.534102 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-token\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.574503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.572905 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hk5h\" (UniqueName: \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-kube-api-access-7hk5h\") pod \"e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:00.696503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.682903 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 09:18:00.696503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.682965 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 09:18:00.822807 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:00.822774 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:01.297509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:01.296766 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6"] Nov 03 09:18:01.317437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:01.313292 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:01.317437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:01.313377 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:01.317437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:01.313421 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:01.684597 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:01.683923 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 03 09:18:01.684597 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:01.683987 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 03 09:18:01.937572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:01.937012 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" event={"ID":"63a8c80d-0637-4e7f-81b9-e17e2523d291","Type":"ContainerStarted","Data":"957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82"} Nov 03 09:18:01.937572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:01.937087 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" event={"ID":"63a8c80d-0637-4e7f-81b9-e17e2523d291","Type":"ContainerStarted","Data":"edb88845d49fcf11990bad2fc5e5ae035f9dfa0531ac7514164ca8b455a93aa9"} Nov 03 09:18:02.073557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:02.066329 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" podStartSLOduration=2.066307222 podStartE2EDuration="2.066307222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:02.062982972 +0000 UTC m=+2314.987925210" watchObservedRunningTime="2026-11-03 09:18:02.066307222 +0000 UTC m=+2314.991249440" Nov 03 09:18:02.822948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:02.822914 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:02.830122 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:02.830091 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:02.976756 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:02.976717 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:02.978403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:02.978377 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:18:03.544510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:03.532875 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:18:03.544510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:03.534140 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:18:03.546102 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:03.545024 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:18:04.995514 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:04.994557 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:18:05.316564 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:05.313586 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded" Nov 03 09:18:07.908357 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:07.908328 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:18:07.908888 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:07.908409 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:18:08.558614 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.558577 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb"] Nov 03 09:18:08.564183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.564154 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:18:08.597065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.597008 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmcmx\" (UniqueName: \"kubernetes.io/projected/1cc5a185-5f49-47f1-a634-65346de5e8b8-kube-api-access-lmcmx\") pod \"payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb\" (UID: \"1cc5a185-5f49-47f1-a634-65346de5e8b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:18:08.597269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.597178 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc5a185-5f49-47f1-a634-65346de5e8b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb\" (UID: \"1cc5a185-5f49-47f1-a634-65346de5e8b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:18:08.620824 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.620791 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-627e5f\"" Nov 03 09:18:08.621028 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.620977 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-627e5f-dockercfg-n4sc7\"" Nov 03 09:18:08.631384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.631353 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb"] Nov 03 09:18:08.726663 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.726598 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmcmx\" (UniqueName: \"kubernetes.io/projected/1cc5a185-5f49-47f1-a634-65346de5e8b8-kube-api-access-lmcmx\") pod \"payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb\" (UID: \"1cc5a185-5f49-47f1-a634-65346de5e8b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:18:08.727681 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.727013 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc5a185-5f49-47f1-a634-65346de5e8b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb\" (UID: \"1cc5a185-5f49-47f1-a634-65346de5e8b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:18:08.728712 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.728370 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc5a185-5f49-47f1-a634-65346de5e8b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb\" (UID: \"1cc5a185-5f49-47f1-a634-65346de5e8b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:18:08.740675 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.740641 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w"] Nov 03 09:18:08.753399 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.753352 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:18:08.805837 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.805800 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w"] Nov 03 09:18:08.806022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.805950 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmcmx\" (UniqueName: \"kubernetes.io/projected/1cc5a185-5f49-47f1-a634-65346de5e8b8-kube-api-access-lmcmx\") pod \"payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb\" (UID: \"1cc5a185-5f49-47f1-a634-65346de5e8b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:18:08.828976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.828879 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w\" (UID: \"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:18:08.829154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.829053 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzk4k\" (UniqueName: \"kubernetes.io/projected/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-kube-api-access-lzk4k\") pod \"payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w\" (UID: \"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:18:08.881345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.881311 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:18:08.930168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.930133 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzk4k\" (UniqueName: \"kubernetes.io/projected/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-kube-api-access-lzk4k\") pod \"payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w\" (UID: \"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:18:08.930651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.930322 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w\" (UID: \"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:18:08.931052 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.931033 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w\" (UID: \"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:18:08.991638 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:08.991583 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzk4k\" (UniqueName: \"kubernetes.io/projected/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-kube-api-access-lzk4k\") pod \"payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w\" (UID: \"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:18:09.086673 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:09.086579 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:18:09.271960 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:09.271931 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb"] Nov 03 09:18:09.272886 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:18:09.272858 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc5a185_5f49_47f1_a634_65346de5e8b8.slice/crio-e254a6f2b8389ab5ab2f92c5174ff7d4947b29b9bf050eb462ebdf3d355bd4e2 WatchSource:0}: Error finding container e254a6f2b8389ab5ab2f92c5174ff7d4947b29b9bf050eb462ebdf3d355bd4e2: Status 404 returned error can't find the container with id e254a6f2b8389ab5ab2f92c5174ff7d4947b29b9bf050eb462ebdf3d355bd4e2 Nov 03 09:18:09.494720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:09.494680 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w"] Nov 03 09:18:10.120029 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:10.119992 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" event={"ID":"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1","Type":"ContainerStarted","Data":"3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82"} Nov 03 09:18:10.120571 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:10.120061 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:18:10.120571 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:10.120084 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" event={"ID":"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1","Type":"ContainerStarted","Data":"6acd33cfcae82503f9304d4cfb69f8d215d35da5b12f25e4b5ca4d6a77f1c132"} Nov 03 09:18:10.122161 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:10.122135 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" event={"ID":"1cc5a185-5f49-47f1-a634-65346de5e8b8","Type":"ContainerStarted","Data":"4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7"} Nov 03 09:18:10.122311 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:10.122181 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:18:10.122311 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:10.122199 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" event={"ID":"1cc5a185-5f49-47f1-a634-65346de5e8b8","Type":"ContainerStarted","Data":"e254a6f2b8389ab5ab2f92c5174ff7d4947b29b9bf050eb462ebdf3d355bd4e2"} Nov 03 09:18:10.239848 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:10.239784 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" podStartSLOduration=2.239768002 podStartE2EDuration="2.239768002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:10.23789745 +0000 UTC m=+2323.162839686" watchObservedRunningTime="2026-11-03 09:18:10.239768002 +0000 UTC m=+2323.164710220" Nov 03 09:18:10.390632 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:10.374149 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" podStartSLOduration=2.374128754 podStartE2EDuration="2.374128754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:10.370886715 +0000 UTC m=+2323.295828970" watchObservedRunningTime="2026-11-03 09:18:10.374128754 +0000 UTC m=+2323.299070972" Nov 03 09:18:11.132207 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:11.132175 5015 generic.go:356] "Generic (PLEG): container finished" podID="7c3c1ad9-a089-4846-9662-db73cd849332" containerID="dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885" exitCode=0 Nov 03 09:18:11.132689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:11.132247 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" event={"ID":"7c3c1ad9-a089-4846-9662-db73cd849332","Type":"ContainerDied","Data":"dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885"} Nov 03 09:18:11.132874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:11.132856 5015 scope.go:122] "RemoveContainer" containerID="dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885" Nov 03 09:18:11.134267 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:11.134237 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:11.134359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:11.134308 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:11.134359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:11.134342 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:12.167139 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:12.167095 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" event={"ID":"7c3c1ad9-a089-4846-9662-db73cd849332","Type":"ContainerStarted","Data":"00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39"} Nov 03 09:18:12.657511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:12.655910 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:18:12.658217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:12.657853 5015 patch_prober.go:29] interesting pod/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 09:18:12.658217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:12.657934 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 09:18:13.647909 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:13.647870 5015 patch_prober.go:29] interesting pod/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 09:18:13.648598 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:13.648545 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 09:18:14.661665 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:14.661613 5015 patch_prober.go:29] interesting pod/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 09:18:14.662110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:14.661694 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 09:18:15.651959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:15.651922 5015 patch_prober.go:29] interesting pod/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 09:18:15.652260 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:15.652232 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 09:18:15.971561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:15.971503 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6"] Nov 03 09:18:15.976887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:15.976862 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:18:16.029169 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.029137 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6"] Nov 03 09:18:16.067505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.058397 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vspc5\" (UniqueName: \"kubernetes.io/projected/2ec048a3-b147-4650-8949-25377a53aa41-kube-api-access-vspc5\") pod \"payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6\" (UID: \"2ec048a3-b147-4650-8949-25377a53aa41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:18:16.067505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.058467 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec048a3-b147-4650-8949-25377a53aa41-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6\" (UID: \"2ec048a3-b147-4650-8949-25377a53aa41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:18:16.072933 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.072891 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-465707-dockercfg-jzcpc\"" Nov 03 09:18:16.172800 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.172753 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vspc5\" (UniqueName: \"kubernetes.io/projected/2ec048a3-b147-4650-8949-25377a53aa41-kube-api-access-vspc5\") pod \"payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6\" (UID: \"2ec048a3-b147-4650-8949-25377a53aa41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:18:16.173090 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.173068 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec048a3-b147-4650-8949-25377a53aa41-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6\" (UID: \"2ec048a3-b147-4650-8949-25377a53aa41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:18:16.173625 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.173599 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-465707\"" Nov 03 09:18:16.184678 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.184637 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec048a3-b147-4650-8949-25377a53aa41-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6\" (UID: \"2ec048a3-b147-4650-8949-25377a53aa41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:18:16.216507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.213626 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c"] Nov 03 09:18:16.224510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.224416 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c"] Nov 03 09:18:16.225597 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.224786 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:18:16.264518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.264463 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vspc5\" (UniqueName: \"kubernetes.io/projected/2ec048a3-b147-4650-8949-25377a53aa41-kube-api-access-vspc5\") pod \"payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6\" (UID: \"2ec048a3-b147-4650-8949-25377a53aa41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:18:16.295363 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.295330 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:18:16.376274 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.376228 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jljkk\" (UniqueName: \"kubernetes.io/projected/6bf84a87-47f3-4c80-a429-564a523574e6-kube-api-access-jljkk\") pod \"payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c\" (UID: \"6bf84a87-47f3-4c80-a429-564a523574e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:18:16.376515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.376326 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf84a87-47f3-4c80-a429-564a523574e6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c\" (UID: \"6bf84a87-47f3-4c80-a429-564a523574e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:18:16.480236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.478181 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljkk\" (UniqueName: \"kubernetes.io/projected/6bf84a87-47f3-4c80-a429-564a523574e6-kube-api-access-jljkk\") pod \"payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c\" (UID: \"6bf84a87-47f3-4c80-a429-564a523574e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:18:16.480236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.478281 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf84a87-47f3-4c80-a429-564a523574e6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c\" (UID: \"6bf84a87-47f3-4c80-a429-564a523574e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:18:16.480236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.479435 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf84a87-47f3-4c80-a429-564a523574e6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c\" (UID: \"6bf84a87-47f3-4c80-a429-564a523574e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:18:16.564163 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.564125 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljkk\" (UniqueName: \"kubernetes.io/projected/6bf84a87-47f3-4c80-a429-564a523574e6-kube-api-access-jljkk\") pod \"payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c\" (UID: \"6bf84a87-47f3-4c80-a429-564a523574e6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:18:16.571914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.571892 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:18:16.649639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.648068 5015 patch_prober.go:29] interesting pod/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 09:18:16.649639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.648125 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 09:18:16.713780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.713686 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6"] Nov 03 09:18:16.902576 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:18:16.902542 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:18:16.973455 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:18:16.973416 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf84a87_47f3_4c80_a429_564a523574e6.slice/crio-ce6d88a9a633654d0b9d9977ff765d8a4708cad94717a76ea380622c54df2617 WatchSource:0}: Error finding container ce6d88a9a633654d0b9d9977ff765d8a4708cad94717a76ea380622c54df2617: Status 404 returned error can't find the container with id ce6d88a9a633654d0b9d9977ff765d8a4708cad94717a76ea380622c54df2617 Nov 03 09:18:16.973929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:16.973838 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c"] Nov 03 09:18:17.239362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:17.239314 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" event={"ID":"2ec048a3-b147-4650-8949-25377a53aa41","Type":"ContainerStarted","Data":"53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8"} Nov 03 09:18:17.239937 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:17.239399 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" event={"ID":"2ec048a3-b147-4650-8949-25377a53aa41","Type":"ContainerStarted","Data":"e1191da052364757a8af13c046ea220807250ca407c4b6af301e3afff34c6a39"} Nov 03 09:18:17.239937 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:17.239592 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:18:17.242088 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:17.242058 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" event={"ID":"6bf84a87-47f3-4c80-a429-564a523574e6","Type":"ContainerStarted","Data":"c8d5f679592cf8ba3db7ec1a718bf50585fe35a20b2dd1761a47111043148ebb"} Nov 03 09:18:17.242223 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:17.242114 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" event={"ID":"6bf84a87-47f3-4c80-a429-564a523574e6","Type":"ContainerStarted","Data":"ce6d88a9a633654d0b9d9977ff765d8a4708cad94717a76ea380622c54df2617"} Nov 03 09:18:17.242223 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:17.242146 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:18:17.358688 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:17.358593 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" podStartSLOduration=2.358578408 podStartE2EDuration="2.358578408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:17.357004539 +0000 UTC m=+2330.281946775" watchObservedRunningTime="2026-11-03 09:18:17.358578408 +0000 UTC m=+2330.283520625" Nov 03 09:18:17.480143 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:17.477396 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" podStartSLOduration=1.477375549 podStartE2EDuration="1.477375549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:17.475001519 +0000 UTC m=+2330.399943763" watchObservedRunningTime="2026-11-03 09:18:17.477375549 +0000 UTC m=+2330.402317767" Nov 03 09:18:17.655402 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:17.655321 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:18:17.655828 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:17.655811 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:18:17.656438 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:17.656423 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:18:18.548966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.548925 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2"] Nov 03 09:18:18.553284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.553263 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:18:18.609650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.609618 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-6f8ed0\"" Nov 03 09:18:18.609885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.609807 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-6f8ed0-dockercfg-7lbbq\"" Nov 03 09:18:18.613012 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.612986 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2"] Nov 03 09:18:18.724500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.720764 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2\" (UID: \"cc657df5-bac1-45a9-a46e-2ed9e2a99e79\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:18:18.724500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.720846 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k756g\" (UniqueName: \"kubernetes.io/projected/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-kube-api-access-k756g\") pod \"payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2\" (UID: \"cc657df5-bac1-45a9-a46e-2ed9e2a99e79\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:18:18.733506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.730199 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm"] Nov 03 09:18:18.738955 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.737963 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:18:18.777129 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.777091 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm"] Nov 03 09:18:18.822544 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.822427 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2\" (UID: \"cc657df5-bac1-45a9-a46e-2ed9e2a99e79\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:18:18.822725 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.822574 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26718843-e793-46f7-a49e-226a38c89794-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm\" (UID: \"26718843-e793-46f7-a49e-226a38c89794\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:18:18.822725 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.822613 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k756g\" (UniqueName: \"kubernetes.io/projected/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-kube-api-access-k756g\") pod \"payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2\" (UID: \"cc657df5-bac1-45a9-a46e-2ed9e2a99e79\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:18:18.822860 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.822782 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssk2j\" (UniqueName: \"kubernetes.io/projected/26718843-e793-46f7-a49e-226a38c89794-kube-api-access-ssk2j\") pod \"payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm\" (UID: \"26718843-e793-46f7-a49e-226a38c89794\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:18:18.823133 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.823111 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2\" (UID: \"cc657df5-bac1-45a9-a46e-2ed9e2a99e79\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:18:18.883718 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.883681 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k756g\" (UniqueName: \"kubernetes.io/projected/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-kube-api-access-k756g\") pod \"payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2\" (UID: \"cc657df5-bac1-45a9-a46e-2ed9e2a99e79\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:18:18.924589 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.924544 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26718843-e793-46f7-a49e-226a38c89794-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm\" (UID: \"26718843-e793-46f7-a49e-226a38c89794\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:18:18.924798 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.924658 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssk2j\" (UniqueName: \"kubernetes.io/projected/26718843-e793-46f7-a49e-226a38c89794-kube-api-access-ssk2j\") pod \"payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm\" (UID: \"26718843-e793-46f7-a49e-226a38c89794\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:18:18.925305 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.925282 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26718843-e793-46f7-a49e-226a38c89794-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm\" (UID: \"26718843-e793-46f7-a49e-226a38c89794\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:18:18.985649 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:18.985620 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssk2j\" (UniqueName: \"kubernetes.io/projected/26718843-e793-46f7-a49e-226a38c89794-kube-api-access-ssk2j\") pod \"payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm\" (UID: \"26718843-e793-46f7-a49e-226a38c89794\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:18:19.066552 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:19.066509 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:18:19.170516 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:19.168015 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:18:19.455648 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:19.455622 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm"] Nov 03 09:18:19.457726 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:18:19.457695 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26718843_e793_46f7_a49e_226a38c89794.slice/crio-7d5cfc6992dc550871e82fdf432da856584d882ec9da44112d717030df4e6ed2 WatchSource:0}: Error finding container 7d5cfc6992dc550871e82fdf432da856584d882ec9da44112d717030df4e6ed2: Status 404 returned error can't find the container with id 7d5cfc6992dc550871e82fdf432da856584d882ec9da44112d717030df4e6ed2 Nov 03 09:18:19.578565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:19.578533 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2"] Nov 03 09:18:20.278579 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:20.278537 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" event={"ID":"cc657df5-bac1-45a9-a46e-2ed9e2a99e79","Type":"ContainerStarted","Data":"6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3"} Nov 03 09:18:20.278796 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:20.278603 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" event={"ID":"cc657df5-bac1-45a9-a46e-2ed9e2a99e79","Type":"ContainerStarted","Data":"1de28f3acbdcaa2c9b92b9cafb9169369b4547631d46e931e99e3427565d11a4"} Nov 03 09:18:20.278796 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:20.278630 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:18:20.280371 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:20.280342 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" event={"ID":"26718843-e793-46f7-a49e-226a38c89794","Type":"ContainerStarted","Data":"1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758"} Nov 03 09:18:20.280460 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:20.280387 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" event={"ID":"26718843-e793-46f7-a49e-226a38c89794","Type":"ContainerStarted","Data":"7d5cfc6992dc550871e82fdf432da856584d882ec9da44112d717030df4e6ed2"} Nov 03 09:18:20.280460 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:20.280438 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:18:20.399809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:20.399757 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" podStartSLOduration=2.399743606 podStartE2EDuration="2.399743606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:20.394624292 +0000 UTC m=+2333.319566510" watchObservedRunningTime="2026-11-03 09:18:20.399743606 +0000 UTC m=+2333.324685852" Nov 03 09:18:20.514759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:20.514709 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" podStartSLOduration=2.514693668 podStartE2EDuration="2.514693668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:20.511445703 +0000 UTC m=+2333.436387919" watchObservedRunningTime="2026-11-03 09:18:20.514693668 +0000 UTC m=+2333.439635884" Nov 03 09:18:21.135971 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:21.135937 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:18:21.136411 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:21.136331 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:18:28.257374 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:28.257344 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:18:28.257890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:28.257419 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:18:29.643125 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.643090 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24"] Nov 03 09:18:29.649234 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.649208 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.703419 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.703379 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24"] Nov 03 09:18:29.748318 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.748283 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-433821-serving-cert\"" Nov 03 09:18:29.794195 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.794160 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmq9j\" (UniqueName: \"kubernetes.io/projected/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-kube-api-access-xmq9j\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.794402 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.794375 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.794597 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.794575 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.794710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.794694 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.896756 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.896646 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.897519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.896760 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.897680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.897557 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmq9j\" (UniqueName: \"kubernetes.io/projected/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-kube-api-access-xmq9j\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.897770 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.897744 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.901826 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.901778 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.904517 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.904449 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.907979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.907947 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.958234 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.958202 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmq9j\" (UniqueName: \"kubernetes.io/projected/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-kube-api-access-xmq9j\") pod \"maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:29.965035 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:29.965004 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:30.357167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:30.357133 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24"] Nov 03 09:18:30.357367 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:18:30.357342 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48572303_fb5d_4f95_bf3b_7ba9e992d5e8.slice/crio-0fd8f510fe84ddc8810e36e89319c11264d278e06b44a3936c7a211807320dba WatchSource:0}: Error finding container 0fd8f510fe84ddc8810e36e89319c11264d278e06b44a3936c7a211807320dba: Status 404 returned error can't find the container with id 0fd8f510fe84ddc8810e36e89319c11264d278e06b44a3936c7a211807320dba Nov 03 09:18:31.291872 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:31.291848 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:18:31.292250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:31.291928 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:18:31.366732 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:31.366683 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" event={"ID":"48572303-fb5d-4f95-bf3b-7ba9e992d5e8","Type":"ContainerStarted","Data":"3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565"} Nov 03 09:18:31.366921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:31.366754 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" event={"ID":"48572303-fb5d-4f95-bf3b-7ba9e992d5e8","Type":"ContainerStarted","Data":"0fd8f510fe84ddc8810e36e89319c11264d278e06b44a3936c7a211807320dba"} Nov 03 09:18:31.366921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:31.366783 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:31.651352 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:31.651297 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" podStartSLOduration=1.799784161 podStartE2EDuration="2.651282672s" totalImagesPullingTime="851.498511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:31.647016487 +0000 UTC m=+2344.571958732" watchObservedRunningTime="2026-11-03 09:18:31.651282672 +0000 UTC m=+2344.576224918" Nov 03 09:18:37.383594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:37.383566 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:18:46.921339 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:18:46.921295 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:18:50.583754 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.583713 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb"] Nov 03 09:18:50.588922 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.588898 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:50.647116 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.647080 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb"] Nov 03 09:18:50.692120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.692091 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-627e5f-serving-cert\"" Nov 03 09:18:50.768218 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.768181 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw284\" (UniqueName: \"kubernetes.io/projected/850f7210-fa1a-409c-b999-c76ffb9de8ad-kube-api-access-gw284\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:50.768424 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.768323 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:50.768424 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.768394 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:50.768597 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.768513 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850f7210-fa1a-409c-b999-c76ffb9de8ad-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:50.870803 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.870705 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:50.871008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.870849 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:50.871085 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.871012 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850f7210-fa1a-409c-b999-c76ffb9de8ad-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:50.871150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.871094 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw284\" (UniqueName: \"kubernetes.io/projected/850f7210-fa1a-409c-b999-c76ffb9de8ad-kube-api-access-gw284\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:50.871339 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:18:50.871316 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-serving-cert: secret "maas-api-e2e-shared-a-w3-627e5f-serving-cert" not found Nov 03 09:18:50.871420 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:18:50.871407 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-maas-api-tls podName:850f7210-fa1a-409c-b999-c76ffb9de8ad nodeName:}" failed. No retries permitted until 2026-11-03 09:18:51.371386676 +0000 UTC m=+2364.296328898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-maas-api-tls") pod "maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" (UID: "850f7210-fa1a-409c-b999-c76ffb9de8ad") : secret "maas-api-e2e-shared-a-w3-627e5f-serving-cert" not found Nov 03 09:18:50.872333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.872307 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850f7210-fa1a-409c-b999-c76ffb9de8ad-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:50.875293 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.875267 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:50.931689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:50.931654 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw284\" (UniqueName: \"kubernetes.io/projected/850f7210-fa1a-409c-b999-c76ffb9de8ad-kube-api-access-gw284\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:51.382191 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:51.381713 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:51.385040 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:51.385016 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:51.504146 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:51.504103 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:51.904617 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:51.904580 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb"] Nov 03 09:18:51.905242 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:18:51.905220 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850f7210_fa1a_409c_b999_c76ffb9de8ad.slice/crio-aa5c65578df03b5952ef1f5899bfd8ae0e5ad8fa8170355f6bfa8547e0003b95 WatchSource:0}: Error finding container aa5c65578df03b5952ef1f5899bfd8ae0e5ad8fa8170355f6bfa8547e0003b95: Status 404 returned error can't find the container with id aa5c65578df03b5952ef1f5899bfd8ae0e5ad8fa8170355f6bfa8547e0003b95 Nov 03 09:18:52.630448 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:52.630394 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" event={"ID":"850f7210-fa1a-409c-b999-c76ffb9de8ad","Type":"ContainerStarted","Data":"aa5c65578df03b5952ef1f5899bfd8ae0e5ad8fa8170355f6bfa8547e0003b95"} Nov 03 09:18:53.640869 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:53.640828 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" event={"ID":"850f7210-fa1a-409c-b999-c76ffb9de8ad","Type":"ContainerStarted","Data":"d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be"} Nov 03 09:18:53.643310 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:53.640908 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:18:53.762371 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:53.762317 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" podStartSLOduration=2.9634877509999997 podStartE2EDuration="3.762302052s" totalImagesPullingTime="798.814301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:53.75911665 +0000 UTC m=+2366.684058864" watchObservedRunningTime="2026-11-03 09:18:53.762302052 +0000 UTC m=+2366.687244268" Nov 03 09:18:54.650459 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.650420 5015 generic.go:356] "Generic (PLEG): container finished" podID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerID="ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6" exitCode=0 Nov 03 09:18:54.650927 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.650528 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" event={"ID":"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d","Type":"ContainerDied","Data":"ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6"} Nov 03 09:18:54.651139 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.651122 5015 scope.go:122] "RemoveContainer" containerID="ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6" Nov 03 09:18:54.652912 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.652879 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:54.653045 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.652956 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:54.653045 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.652993 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:54.848468 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.848437 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96"] Nov 03 09:18:54.853255 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.853232 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:54.917056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.916961 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96"] Nov 03 09:18:54.939823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.939784 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:54.940015 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.939833 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d233c76-e29e-43a0-abd0-212ce5df00a3-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:54.940015 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.939948 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:54.940145 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.940016 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:54.940145 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.940039 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmmvn\" (UniqueName: \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-kube-api-access-nmmvn\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:54.940257 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.940165 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:54.940257 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.940221 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:54.940358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.940301 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:54.940412 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:54.940398 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.041972 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.041934 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.042162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.042009 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.042162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.042035 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmmvn\" (UniqueName: \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-kube-api-access-nmmvn\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.042162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.042090 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.042322 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.042294 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.042445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.042420 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.042628 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.042447 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.042708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.042634 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.042768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.042718 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.042768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.042731 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.042768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.042741 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d233c76-e29e-43a0-abd0-212ce5df00a3-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.043220 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.043195 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.043360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.043334 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.043463 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.043428 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d233c76-e29e-43a0-abd0-212ce5df00a3-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.044954 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.044916 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.046819 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.046798 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.105733 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.105701 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.106031 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.105997 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmmvn\" (UniqueName: \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-kube-api-access-nmmvn\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.180554 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.178930 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:55.659499 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:18:55.653352 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d233c76_e29e_43a0_abd0_212ce5df00a3.slice/crio-d732f0405c1699c78773fb5356e4ba458670c700e308730bea8605a2e7e50377 WatchSource:0}: Error finding container d732f0405c1699c78773fb5356e4ba458670c700e308730bea8605a2e7e50377: Status 404 returned error can't find the container with id d732f0405c1699c78773fb5356e4ba458670c700e308730bea8605a2e7e50377 Nov 03 09:18:55.675140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.674223 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" event={"ID":"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d","Type":"ContainerStarted","Data":"c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091"} Nov 03 09:18:55.675140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.674434 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:55.675140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.674566 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:55.675140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.674616 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:18:55.676311 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:55.675692 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96"] Nov 03 09:18:56.561510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:56.558881 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:18:56.561510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:56.559009 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 09:18:56.561510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:56.559053 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 09:18:56.704629 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:56.704595 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" event={"ID":"5d233c76-e29e-43a0-abd0-212ce5df00a3","Type":"ContainerStarted","Data":"81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592"} Nov 03 09:18:56.705058 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:56.704654 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" event={"ID":"5d233c76-e29e-43a0-abd0-212ce5df00a3","Type":"ContainerStarted","Data":"d732f0405c1699c78773fb5356e4ba458670c700e308730bea8605a2e7e50377"} Nov 03 09:18:56.832886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:56.832775 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" podStartSLOduration=2.832761534 podStartE2EDuration="2.832761534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:56.829633644 +0000 UTC m=+2369.754575878" watchObservedRunningTime="2026-11-03 09:18:56.832761534 +0000 UTC m=+2369.757703751" Nov 03 09:18:57.181429 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:57.181340 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:18:57.181834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:57.181806 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 09:18:57.182034 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:57.182009 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 09:18:57.561512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:57.559783 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 09:18:57.561512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:57.559843 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 09:18:58.183060 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:58.183018 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 09:18:58.183704 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:58.183672 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 09:18:58.548924 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:58.548883 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 09:18:58.549286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:58.549257 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 09:18:59.181512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:59.179944 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 09:18:59.181512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:59.180066 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 09:18:59.548897 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:59.548859 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 09:18:59.549563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:59.549531 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 09:18:59.679723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:18:59.679691 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:19:00.180501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:00.179916 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 09:19:00.180501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:00.180000 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 09:19:00.552917 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:00.552882 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 09:19:00.553576 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:00.553542 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 09:19:01.179260 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.179220 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 09:19:01.179507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.179299 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 09:19:01.564790 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.564758 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:19:01.566852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.566830 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:19:01.571798 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.571773 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:19:01.674250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.674218 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk"] Nov 03 09:19:01.680077 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.680052 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:01.693044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.693014 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk"] Nov 03 09:19:01.773038 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.772980 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-465707-serving-cert\"" Nov 03 09:19:01.826044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.825962 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:01.826186 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.826169 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b5cc05-6c87-427d-8691-8bfe297128a7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:01.826257 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.826239 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rm8q\" (UniqueName: \"kubernetes.io/projected/e7b5cc05-6c87-427d-8691-8bfe297128a7-kube-api-access-8rm8q\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:01.826302 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.826290 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:01.936744 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.934786 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:01.936744 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.935125 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b5cc05-6c87-427d-8691-8bfe297128a7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:01.936744 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.935275 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rm8q\" (UniqueName: \"kubernetes.io/projected/e7b5cc05-6c87-427d-8691-8bfe297128a7-kube-api-access-8rm8q\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:01.936744 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.935344 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:01.936744 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:19:01.935590 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-serving-cert: secret "maas-api-e2e-shared-a-w1-465707-serving-cert" not found Nov 03 09:19:01.936744 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:19:01.935672 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-maas-api-tls podName:e7b5cc05-6c87-427d-8691-8bfe297128a7 nodeName:}" failed. No retries permitted until 2026-11-03 09:19:02.435649033 +0000 UTC m=+2375.360591247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-maas-api-tls") pod "maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" (UID: "e7b5cc05-6c87-427d-8691-8bfe297128a7") : secret "maas-api-e2e-shared-a-w1-465707-serving-cert" not found Nov 03 09:19:01.952877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.952836 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b5cc05-6c87-427d-8691-8bfe297128a7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:01.985657 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:01.985533 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:02.015085 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:02.015033 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rm8q\" (UniqueName: \"kubernetes.io/projected/e7b5cc05-6c87-427d-8691-8bfe297128a7-kube-api-access-8rm8q\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:02.179800 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:02.179691 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 09:19:02.179800 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:02.179781 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 09:19:02.443176 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:02.443074 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:02.446734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:02.446693 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:02.602978 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:02.602933 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:03.000025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:02.999994 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk"] Nov 03 09:19:03.004403 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:19:03.004375 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b5cc05_6c87_427d_8691_8bfe297128a7.slice/crio-6a8c1e6d2240f66f53d628680a33aadb01fefcb6f0f295341f0e73b9241d21b6 WatchSource:0}: Error finding container 6a8c1e6d2240f66f53d628680a33aadb01fefcb6f0f295341f0e73b9241d21b6: Status 404 returned error can't find the container with id 6a8c1e6d2240f66f53d628680a33aadb01fefcb6f0f295341f0e73b9241d21b6 Nov 03 09:19:03.296510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:03.296384 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:19:03.297128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:03.297101 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:19:03.297841 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:03.297815 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:19:03.801323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:03.801289 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" event={"ID":"e7b5cc05-6c87-427d-8691-8bfe297128a7","Type":"ContainerStarted","Data":"6a8c1e6d2240f66f53d628680a33aadb01fefcb6f0f295341f0e73b9241d21b6"} Nov 03 09:19:04.811829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:04.811791 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" event={"ID":"e7b5cc05-6c87-427d-8691-8bfe297128a7","Type":"ContainerStarted","Data":"5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639"} Nov 03 09:19:04.812288 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:04.811882 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:04.931634 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:04.931571 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" podStartSLOduration=3.161168522 podStartE2EDuration="3.93155698s" totalImagesPullingTime="770.388458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:19:04.929458279 +0000 UTC m=+2377.854400515" watchObservedRunningTime="2026-11-03 09:19:04.93155698 +0000 UTC m=+2377.856499197" Nov 03 09:19:10.828295 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:10.828263 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:19:16.902363 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:19:16.902326 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:19:17.956128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:17.956034 5015 generic.go:356] "Generic (PLEG): container finished" podID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerID="957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82" exitCode=0 Nov 03 09:19:17.956128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:17.956102 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" event={"ID":"63a8c80d-0637-4e7f-81b9-e17e2523d291","Type":"ContainerDied","Data":"957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82"} Nov 03 09:19:17.956632 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:17.956547 5015 scope.go:122] "RemoveContainer" containerID="957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82" Nov 03 09:19:17.958446 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:17.958408 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:19:17.958615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:17.958521 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:19:17.958615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:17.958571 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:19:17.977318 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:17.977294 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:19:18.969403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:18.969369 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" event={"ID":"63a8c80d-0637-4e7f-81b9-e17e2523d291","Type":"ContainerStarted","Data":"79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05"} Nov 03 09:19:19.823074 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:19.823037 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:19:19.824169 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:19.824143 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 09:19:19.824298 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:19.824223 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 09:19:20.823974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:20.823940 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 09:19:20.824419 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:20.824002 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 09:19:21.823330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:21.823290 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 09:19:21.823631 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:21.823358 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 09:19:22.823717 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:22.823679 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 09:19:22.824146 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:22.823747 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 09:19:23.823881 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:23.823843 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 09:19:23.824286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:23.823909 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 09:19:24.824071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:24.824022 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 09:19:24.824497 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:24.824132 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 09:19:25.828015 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:25.827983 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:19:25.828531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:25.828512 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:19:25.829301 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:25.829279 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:19:27.646545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:27.646514 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:19:27.658808 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:27.658785 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:19:46.902769 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:19:46.902727 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:19:49.000845 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.000810 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7"] Nov 03 09:19:49.006687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.006665 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.061217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.061187 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-f9b517-serving-cert\"" Nov 03 09:19:49.115511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.115438 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7"] Nov 03 09:19:49.122785 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.122757 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh8lw\" (UniqueName: \"kubernetes.io/projected/a39c892d-ff3a-43d4-bcad-c77218158498-kube-api-access-fh8lw\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.122945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.122820 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.123019 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.123000 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39c892d-ff3a-43d4-bcad-c77218158498-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.123139 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.123117 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.224141 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.224102 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.224334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.224237 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh8lw\" (UniqueName: \"kubernetes.io/projected/a39c892d-ff3a-43d4-bcad-c77218158498-kube-api-access-fh8lw\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.224334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.224281 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.224334 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:19:49.224238 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-serving-cert: secret "maas-api-e2e-shared-a-w0-f9b517-serving-cert" not found Nov 03 09:19:49.224562 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:19:49.224394 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-maas-api-tls podName:a39c892d-ff3a-43d4-bcad-c77218158498 nodeName:}" failed. No retries permitted until 2026-11-03 09:19:49.724373082 +0000 UTC m=+2422.649315280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-maas-api-tls") pod "maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" (UID: "a39c892d-ff3a-43d4-bcad-c77218158498") : secret "maas-api-e2e-shared-a-w0-f9b517-serving-cert" not found Nov 03 09:19:49.224562 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.224428 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39c892d-ff3a-43d4-bcad-c77218158498-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.225086 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.225066 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39c892d-ff3a-43d4-bcad-c77218158498-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.227136 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.227113 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.284464 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.284385 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh8lw\" (UniqueName: \"kubernetes.io/projected/a39c892d-ff3a-43d4-bcad-c77218158498-kube-api-access-fh8lw\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.731576 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.731542 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.734214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.734189 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:49.921062 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:49.921026 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:50.314732 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:50.314700 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7"] Nov 03 09:19:50.315255 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:19:50.315231 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39c892d_ff3a_43d4_bcad_c77218158498.slice/crio-930d867497ca88a8a5c32ba334a9cb806761fc689eae6c6ee2bfae1dc3a39da0 WatchSource:0}: Error finding container 930d867497ca88a8a5c32ba334a9cb806761fc689eae6c6ee2bfae1dc3a39da0: Status 404 returned error can't find the container with id 930d867497ca88a8a5c32ba334a9cb806761fc689eae6c6ee2bfae1dc3a39da0 Nov 03 09:19:51.251014 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:51.250974 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" event={"ID":"a39c892d-ff3a-43d4-bcad-c77218158498","Type":"ContainerStarted","Data":"0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f"} Nov 03 09:19:51.251191 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:51.251042 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" event={"ID":"a39c892d-ff3a-43d4-bcad-c77218158498","Type":"ContainerStarted","Data":"930d867497ca88a8a5c32ba334a9cb806761fc689eae6c6ee2bfae1dc3a39da0"} Nov 03 09:19:51.251191 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:51.251073 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:19:51.376104 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:51.375993 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" podStartSLOduration=2.580579275 podStartE2EDuration="3.375976522s" totalImagesPullingTime="795.397247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:19:51.373953955 +0000 UTC m=+2424.298896199" watchObservedRunningTime="2026-11-03 09:19:51.375976522 +0000 UTC m=+2424.300918738" Nov 03 09:19:57.268837 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:19:57.268804 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:20:14.098845 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.098811 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd"] Nov 03 09:20:14.104452 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.104429 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:20:14.142819 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.142787 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qsnz\" (UniqueName: \"kubernetes.io/projected/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-kube-api-access-6qsnz\") pod \"payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd\" (UID: \"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:20:14.143055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.143028 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd\" (UID: \"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:20:14.160898 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.160865 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd"] Nov 03 09:20:14.205879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.205821 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-b196c8-dockercfg-2l6c8\"" Nov 03 09:20:14.205879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.205826 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b196c8\"" Nov 03 09:20:14.254502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.245967 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qsnz\" (UniqueName: \"kubernetes.io/projected/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-kube-api-access-6qsnz\") pod \"payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd\" (UID: \"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:20:14.254502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.246147 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd\" (UID: \"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:20:14.254502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.247255 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd\" (UID: \"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:20:14.302497 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.301731 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5"] Nov 03 09:20:14.321551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.321503 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:20:14.382356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.356202 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qsnz\" (UniqueName: \"kubernetes.io/projected/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-kube-api-access-6qsnz\") pod \"payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd\" (UID: \"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:20:14.385722 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.385691 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5"] Nov 03 09:20:14.418828 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.418806 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:20:14.451821 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.451786 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqfw\" (UniqueName: \"kubernetes.io/projected/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-kube-api-access-2vqfw\") pod \"payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5\" (UID: \"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:20:14.451938 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.451886 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5\" (UID: \"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:20:14.553055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.552830 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqfw\" (UniqueName: \"kubernetes.io/projected/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-kube-api-access-2vqfw\") pod \"payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5\" (UID: \"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:20:14.553055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.552901 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5\" (UID: \"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:20:14.554045 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.554017 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5\" (UID: \"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:20:14.612331 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.612302 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqfw\" (UniqueName: \"kubernetes.io/projected/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-kube-api-access-2vqfw\") pod \"payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5\" (UID: \"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:20:14.700692 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.700617 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:20:14.825734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:14.825656 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd"] Nov 03 09:20:15.095216 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:15.095185 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5"] Nov 03 09:20:15.095348 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:20:15.095300 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494d276c_7e71_4d57_a8b7_8cfc32b6b4c7.slice/crio-0e11ca6579784913c054c070756640f34b185dcbda34ea58d5a4e3e0eb03d6f0 WatchSource:0}: Error finding container 0e11ca6579784913c054c070756640f34b185dcbda34ea58d5a4e3e0eb03d6f0: Status 404 returned error can't find the container with id 0e11ca6579784913c054c070756640f34b185dcbda34ea58d5a4e3e0eb03d6f0 Nov 03 09:20:15.470895 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:15.470624 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" event={"ID":"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f","Type":"ContainerStarted","Data":"1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3"} Nov 03 09:20:15.472874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:15.471464 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" event={"ID":"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f","Type":"ContainerStarted","Data":"45c84b959a4aaab70bc6ab89e8dcfe469e9f88adf79d7ce62d70090128d1f2cc"} Nov 03 09:20:15.473341 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:15.473322 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:20:15.476003 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:15.475979 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" event={"ID":"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7","Type":"ContainerStarted","Data":"79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca"} Nov 03 09:20:15.476169 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:15.476154 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" event={"ID":"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7","Type":"ContainerStarted","Data":"0e11ca6579784913c054c070756640f34b185dcbda34ea58d5a4e3e0eb03d6f0"} Nov 03 09:20:15.477071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:15.477022 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:20:15.590524 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:15.590437 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" podStartSLOduration=1.590424243 podStartE2EDuration="1.590424243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:20:15.58937627 +0000 UTC m=+2448.514318506" watchObservedRunningTime="2026-11-03 09:20:15.590424243 +0000 UTC m=+2448.515366459" Nov 03 09:20:15.719709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:15.719643 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" podStartSLOduration=1.719624192 podStartE2EDuration="1.719624192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:20:15.705356944 +0000 UTC m=+2448.630299160" watchObservedRunningTime="2026-11-03 09:20:15.719624192 +0000 UTC m=+2448.644566409" Nov 03 09:20:16.903444 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:20:16.903407 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:20:26.262063 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.262026 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs"] Nov 03 09:20:26.266829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.266806 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:20:26.327885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.327848 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs"] Nov 03 09:20:26.337240 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.337211 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7667l\" (UniqueName: \"kubernetes.io/projected/7b9401bf-3719-4419-a9f1-658e6ba190d1-kube-api-access-7667l\") pod \"payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs\" (UID: \"7b9401bf-3719-4419-a9f1-658e6ba190d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:20:26.337694 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.337670 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9401bf-3719-4419-a9f1-658e6ba190d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs\" (UID: \"7b9401bf-3719-4419-a9f1-658e6ba190d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:20:26.406180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.406144 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-3df1cc-dockercfg-pkqg5\"" Nov 03 09:20:26.408552 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.406537 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-3df1cc\"" Nov 03 09:20:26.465204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.460691 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7667l\" (UniqueName: \"kubernetes.io/projected/7b9401bf-3719-4419-a9f1-658e6ba190d1-kube-api-access-7667l\") pod \"payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs\" (UID: \"7b9401bf-3719-4419-a9f1-658e6ba190d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:20:26.486543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.486512 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9401bf-3719-4419-a9f1-658e6ba190d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs\" (UID: \"7b9401bf-3719-4419-a9f1-658e6ba190d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:20:26.487781 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.487751 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9401bf-3719-4419-a9f1-658e6ba190d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs\" (UID: \"7b9401bf-3719-4419-a9f1-658e6ba190d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:20:26.547788 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.547706 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7667l\" (UniqueName: \"kubernetes.io/projected/7b9401bf-3719-4419-a9f1-658e6ba190d1-kube-api-access-7667l\") pod \"payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs\" (UID: \"7b9401bf-3719-4419-a9f1-658e6ba190d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:20:26.584494 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.584439 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:20:26.984833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:26.984800 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs"] Nov 03 09:20:26.985537 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:20:26.985466 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9401bf_3719_4419_a9f1_658e6ba190d1.slice/crio-65165101b26e0c193ab50af614704412f5a7eadefb42f653c8dc96bfc4f53240 WatchSource:0}: Error finding container 65165101b26e0c193ab50af614704412f5a7eadefb42f653c8dc96bfc4f53240: Status 404 returned error can't find the container with id 65165101b26e0c193ab50af614704412f5a7eadefb42f653c8dc96bfc4f53240 Nov 03 09:20:27.503839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:27.503804 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:20:27.505283 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:27.505218 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:20:27.578003 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:27.577970 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" event={"ID":"7b9401bf-3719-4419-a9f1-658e6ba190d1","Type":"ContainerStarted","Data":"65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586"} Nov 03 09:20:27.578180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:27.578019 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" event={"ID":"7b9401bf-3719-4419-a9f1-658e6ba190d1","Type":"ContainerStarted","Data":"65165101b26e0c193ab50af614704412f5a7eadefb42f653c8dc96bfc4f53240"} Nov 03 09:20:27.578180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:27.578078 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:20:27.857375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:27.857257 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" podStartSLOduration=1.857238922 podStartE2EDuration="1.857238922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:20:27.855447298 +0000 UTC m=+2460.780389515" watchObservedRunningTime="2026-11-03 09:20:27.857238922 +0000 UTC m=+2460.782181137" Nov 03 09:20:38.595595 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:38.595515 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:20:41.877011 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:41.876972 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r"] Nov 03 09:20:41.877513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:41.877206 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" podUID="10994f28-5c15-4e2d-b97a-40680160270d" containerName="maas-api" containerID="cri-o://ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c" gracePeriod=30 Nov 03 09:20:42.205635 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.205605 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:20:42.331993 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.331907 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10994f28-5c15-4e2d-b97a-40680160270d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10994f28-5c15-4e2d-b97a-40680160270d-openshift-service-ca\") pod \"10994f28-5c15-4e2d-b97a-40680160270d\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " Nov 03 09:20:42.332195 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.332141 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10994f28-5c15-4e2d-b97a-40680160270d-kube-api-access-7fplm\" (UniqueName: \"kubernetes.io/projected/10994f28-5c15-4e2d-b97a-40680160270d-kube-api-access-7fplm\") pod \"10994f28-5c15-4e2d-b97a-40680160270d\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " Nov 03 09:20:42.332260 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.332205 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-metrics-certs\") pod \"10994f28-5c15-4e2d-b97a-40680160270d\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " Nov 03 09:20:42.332260 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.332234 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-maas-api-tls\") pod \"10994f28-5c15-4e2d-b97a-40680160270d\" (UID: \"10994f28-5c15-4e2d-b97a-40680160270d\") " Nov 03 09:20:42.332360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.332294 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10994f28-5c15-4e2d-b97a-40680160270d-openshift-service-ca" pod "10994f28-5c15-4e2d-b97a-40680160270d" (UID: "10994f28-5c15-4e2d-b97a-40680160270d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:20:42.337541 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.334712 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10994f28-5c15-4e2d-b97a-40680160270d-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:42.337854 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.337825 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-maas-api-tls" pod "10994f28-5c15-4e2d-b97a-40680160270d" (UID: "10994f28-5c15-4e2d-b97a-40680160270d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:20:42.342582 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.339153 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-metrics-certs" pod "10994f28-5c15-4e2d-b97a-40680160270d" (UID: "10994f28-5c15-4e2d-b97a-40680160270d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:20:42.342582 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.341022 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10994f28-5c15-4e2d-b97a-40680160270d-kube-api-access-7fplm" pod "10994f28-5c15-4e2d-b97a-40680160270d" (UID: "10994f28-5c15-4e2d-b97a-40680160270d"). InnerVolumeSpecName "kube-api-access-7fplm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:20:42.440919 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.440710 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fplm\" (UniqueName: \"kubernetes.io/projected/10994f28-5c15-4e2d-b97a-40680160270d-kube-api-access-7fplm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:42.440919 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.440748 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:42.440919 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.440767 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10994f28-5c15-4e2d-b97a-40680160270d-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:42.704817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.704725 5015 generic.go:356] "Generic (PLEG): container finished" podID="10994f28-5c15-4e2d-b97a-40680160270d" containerID="ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c" exitCode=0 Nov 03 09:20:42.704817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.704783 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" Nov 03 09:20:42.704817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.704803 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" event={"ID":"10994f28-5c15-4e2d-b97a-40680160270d","Type":"ContainerDied","Data":"ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c"} Nov 03 09:20:42.705130 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.704860 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r" event={"ID":"10994f28-5c15-4e2d-b97a-40680160270d","Type":"ContainerDied","Data":"9e65a92a3158061a0e5a31e935b2625ee24492907d87730a22f83b6c422211c2"} Nov 03 09:20:42.705130 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.704886 5015 scope.go:122] "RemoveContainer" containerID="ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c" Nov 03 09:20:42.716444 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.716424 5015 scope.go:122] "RemoveContainer" containerID="ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c" Nov 03 09:20:42.716775 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:20:42.716755 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c\": container with ID starting with ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c not found: ID does not exist" containerID="ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c" Nov 03 09:20:42.716855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.716785 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c"} err="failed to get container status \"ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c\": rpc error: code = NotFound desc = could not find container \"ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c\": container with ID starting with ed747e0adb8618c132f17111f6ba4d55ac64aff56daf774f340681b52d01f91c not found: ID does not exist" Nov 03 09:20:42.930502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.927165 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r"] Nov 03 09:20:42.950519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:42.950455 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1ce7a7-7b47584867-xwd7r"] Nov 03 09:20:43.211331 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:43.211296 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10994f28-5c15-4e2d-b97a-40680160270d" path="/var/lib/kubelet/pods/10994f28-5c15-4e2d-b97a-40680160270d/volumes" Nov 03 09:20:46.903957 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:20:46.903918 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:20:53.070839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:53.070789 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx"] Nov 03 09:20:53.071398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:53.071085 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" containerID="cri-o://00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39" gracePeriod=30 Nov 03 09:20:58.364549 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.364522 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:20:58.402530 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.402493 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3c1ad9-a089-4846-9662-db73cd849332-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3c1ad9-a089-4846-9662-db73cd849332-istiod-ca-cert\") pod \"7c3c1ad9-a089-4846-9662-db73cd849332\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " Nov 03 09:20:58.402694 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.402540 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c3c1ad9-a089-4846-9662-db73cd849332-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3c1ad9-a089-4846-9662-db73cd849332-istio-podinfo\") pod \"7c3c1ad9-a089-4846-9662-db73cd849332\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " Nov 03 09:20:58.402694 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.402639 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-envoy\") pod \"7c3c1ad9-a089-4846-9662-db73cd849332\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " Nov 03 09:20:58.402798 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.402723 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-certs\") pod \"7c3c1ad9-a089-4846-9662-db73cd849332\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " Nov 03 09:20:58.402853 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.402835 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-credential-socket\") pod \"7c3c1ad9-a089-4846-9662-db73cd849332\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " Nov 03 09:20:58.402904 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.402890 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-istio-token\" (UniqueName: \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-istio-token\") pod \"7c3c1ad9-a089-4846-9662-db73cd849332\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " Nov 03 09:20:58.402960 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.402924 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-data\") pod \"7c3c1ad9-a089-4846-9662-db73cd849332\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " Nov 03 09:20:58.402960 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.402941 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-certs" pod "7c3c1ad9-a089-4846-9662-db73cd849332" (UID: "7c3c1ad9-a089-4846-9662-db73cd849332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:20:58.403075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.402949 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-kube-api-access-zptt8\" (UniqueName: \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-kube-api-access-zptt8\") pod \"7c3c1ad9-a089-4846-9662-db73cd849332\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " Nov 03 09:20:58.403075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.402898 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3c1ad9-a089-4846-9662-db73cd849332-istiod-ca-cert" pod "7c3c1ad9-a089-4846-9662-db73cd849332" (UID: "7c3c1ad9-a089-4846-9662-db73cd849332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:20:58.403075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.403016 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-socket\") pod \"7c3c1ad9-a089-4846-9662-db73cd849332\" (UID: \"7c3c1ad9-a089-4846-9662-db73cd849332\") " Nov 03 09:20:58.403241 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.403199 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-data" pod "7c3c1ad9-a089-4846-9662-db73cd849332" (UID: "7c3c1ad9-a089-4846-9662-db73cd849332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:20:58.403437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.403360 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-socket" pod "7c3c1ad9-a089-4846-9662-db73cd849332" (UID: "7c3c1ad9-a089-4846-9662-db73cd849332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:20:58.403437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.403125 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-credential-socket" pod "7c3c1ad9-a089-4846-9662-db73cd849332" (UID: "7c3c1ad9-a089-4846-9662-db73cd849332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:20:58.404076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.404047 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:58.404173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.404080 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:58.404173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.404096 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:58.404173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.404110 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:58.404173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.404126 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3c1ad9-a089-4846-9662-db73cd849332-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:58.405173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.405136 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-envoy" pod "7c3c1ad9-a089-4846-9662-db73cd849332" (UID: "7c3c1ad9-a089-4846-9662-db73cd849332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:20:58.405283 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.405231 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c3c1ad9-a089-4846-9662-db73cd849332-istio-podinfo" pod "7c3c1ad9-a089-4846-9662-db73cd849332" (UID: "7c3c1ad9-a089-4846-9662-db73cd849332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:20:58.405283 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.405275 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-kube-api-access-zptt8" pod "7c3c1ad9-a089-4846-9662-db73cd849332" (UID: "7c3c1ad9-a089-4846-9662-db73cd849332"). InnerVolumeSpecName "kube-api-access-zptt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:20:58.405444 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.405429 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-istio-token" pod "7c3c1ad9-a089-4846-9662-db73cd849332" (UID: "7c3c1ad9-a089-4846-9662-db73cd849332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:20:58.505753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.505716 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3c1ad9-a089-4846-9662-db73cd849332-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:58.505753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.505745 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:58.505967 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.505760 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zptt8\" (UniqueName: \"kubernetes.io/projected/7c3c1ad9-a089-4846-9662-db73cd849332-kube-api-access-zptt8\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:58.505967 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.505777 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3c1ad9-a089-4846-9662-db73cd849332-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:58.873185 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.873145 5015 generic.go:356] "Generic (PLEG): container finished" podID="7c3c1ad9-a089-4846-9662-db73cd849332" containerID="00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39" exitCode=0 Nov 03 09:20:58.873950 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.873921 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" event={"ID":"7c3c1ad9-a089-4846-9662-db73cd849332","Type":"ContainerDied","Data":"00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39"} Nov 03 09:20:58.874211 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.874193 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" event={"ID":"7c3c1ad9-a089-4846-9662-db73cd849332","Type":"ContainerDied","Data":"366439aa7d4be0b29e929558809791e5681d519d0e167508314d49c66e25f33c"} Nov 03 09:20:58.874346 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.874332 5015 scope.go:122] "RemoveContainer" containerID="00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39" Nov 03 09:20:58.874719 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.874697 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx" Nov 03 09:20:58.960670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.959660 5015 scope.go:122] "RemoveContainer" containerID="dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885" Nov 03 09:20:58.975242 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.975215 5015 scope.go:122] "RemoveContainer" containerID="00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39" Nov 03 09:20:58.975599 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:20:58.975579 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39\": container with ID starting with 00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39 not found: ID does not exist" containerID="00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39" Nov 03 09:20:58.975711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.975610 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39"} err="failed to get container status \"00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39\": rpc error: code = NotFound desc = could not find container \"00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39\": container with ID starting with 00347a8d2bc2798dbb9abe4bd758383aab0d26f8e6cbd8d593de17adce43be39 not found: ID does not exist" Nov 03 09:20:58.975711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.975633 5015 scope.go:122] "RemoveContainer" containerID="dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885" Nov 03 09:20:58.975978 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:20:58.975950 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885\": container with ID starting with dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885 not found: ID does not exist" containerID="dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885" Nov 03 09:20:58.976019 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:58.975991 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885"} err="failed to get container status \"dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885\": rpc error: code = NotFound desc = could not find container \"dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885\": container with ID starting with dae2120e4959e6e7668619347ac34445ffca4ca29f26452c5f6b6901df83a885 not found: ID does not exist" Nov 03 09:20:59.081019 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:59.080977 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx"] Nov 03 09:20:59.128968 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:59.128896 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-1ce7a7-openshift-default-5fb47b6db5-ngxqx"] Nov 03 09:20:59.206200 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:20:59.206160 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" path="/var/lib/kubelet/pods/7c3c1ad9-a089-4846-9662-db73cd849332/volumes" Nov 03 09:21:01.511698 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:01.511663 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm"] Nov 03 09:21:01.626308 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:01.626272 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2"] Nov 03 09:21:03.637109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.637063 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct"] Nov 03 09:21:03.637847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.637827 5015 cpu_manager.go:414] "containerMap: removing container" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" Nov 03 09:21:03.637847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.637847 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" Nov 03 09:21:03.638007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.637863 5015 cpu_manager.go:414] "containerMap: removing container" podUID="10994f28-5c15-4e2d-b97a-40680160270d" containerName="maas-api" Nov 03 09:21:03.638007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.637868 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10994f28-5c15-4e2d-b97a-40680160270d" containerName="maas-api" Nov 03 09:21:03.638007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.637885 5015 cpu_manager.go:414] "containerMap: removing container" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" Nov 03 09:21:03.638007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.637891 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" Nov 03 09:21:03.638208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.638014 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" Nov 03 09:21:03.638208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.638029 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3c1ad9-a089-4846-9662-db73cd849332" containerName="istio-proxy" Nov 03 09:21:03.638208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.638038 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="10994f28-5c15-4e2d-b97a-40680160270d" containerName="maas-api" Nov 03 09:21:03.643174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.643148 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.698008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.697977 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9ed110-openshift-default-dockercfg-5w7tj\"" Nov 03 09:21:03.752326 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.752295 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct"] Nov 03 09:21:03.795681 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.795633 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-credential-socket\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.795681 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.795690 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-certs\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.795945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.795810 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.795945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.795865 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-socket\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.796067 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.796043 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8ps\" (UniqueName: \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-kube-api-access-8w8ps\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.796167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.796149 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-data\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.796229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.796191 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-podinfo\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.796281 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.796242 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-envoy\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.796325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.796290 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-token\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.897659 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.897561 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-envoy\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.897907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.897885 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-token\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.898075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.898057 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-credential-socket\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.898215 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.898197 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-certs\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.898374 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.898359 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.898516 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.898501 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-socket\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.898669 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.898637 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-certs\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.898772 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.898375 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-credential-socket\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.899110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.899066 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.899387 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.899326 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-socket\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.899556 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.899420 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8ps\" (UniqueName: \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-kube-api-access-8w8ps\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.899657 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.899542 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-data\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.899657 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.899607 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-podinfo\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.900262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.900226 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-data\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.904766 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.904737 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-envoy\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.910432 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.910399 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-podinfo\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.958401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.958366 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-token\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:03.959817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:03.959741 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8ps\" (UniqueName: \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-kube-api-access-8w8ps\") pod \"e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:04.258607 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:04.258565 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:04.698243 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:21:04.698206 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e9349e_0cdb_4aa1_ab1b_f44b5dd0ae1f.slice/crio-d2e5a5b2efe5a5e5d000eaafb28bd87a56da3c8bdede84fe8a969829134b1ad4 WatchSource:0}: Error finding container d2e5a5b2efe5a5e5d000eaafb28bd87a56da3c8bdede84fe8a969829134b1ad4: Status 404 returned error can't find the container with id d2e5a5b2efe5a5e5d000eaafb28bd87a56da3c8bdede84fe8a969829134b1ad4 Nov 03 09:21:04.698624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:04.698465 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct"] Nov 03 09:21:04.700876 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:04.700848 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:21:04.700978 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:04.700912 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:21:04.700978 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:04.700943 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:21:04.928488 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:04.928422 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" event={"ID":"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f","Type":"ContainerStarted","Data":"77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb"} Nov 03 09:21:04.928679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:04.928533 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" event={"ID":"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f","Type":"ContainerStarted","Data":"d2e5a5b2efe5a5e5d000eaafb28bd87a56da3c8bdede84fe8a969829134b1ad4"} Nov 03 09:21:05.049796 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:05.049728 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" podStartSLOduration=2.049710088 podStartE2EDuration="2.049710088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:21:05.047842755 +0000 UTC m=+2497.972784993" watchObservedRunningTime="2026-11-03 09:21:05.049710088 +0000 UTC m=+2497.974652306" Nov 03 09:21:05.258765 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:05.258727 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:05.264777 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:05.264755 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:05.940014 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:05.939982 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:05.941082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:05.941063 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:21:06.512695 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.512656 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" podUID="26718843-e793-46f7-a49e-226a38c89794" containerName="payload-processing" containerID="cri-o://1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758" gracePeriod=25 Nov 03 09:21:06.627557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.627516 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" podUID="cc657df5-bac1-45a9-a46e-2ed9e2a99e79" containerName="payload-pre-processing" containerID="cri-o://6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3" gracePeriod=25 Nov 03 09:21:06.799147 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.799113 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:21:06.933236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.933204 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:21:06.944976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.944941 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26718843-e793-46f7-a49e-226a38c89794-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26718843-e793-46f7-a49e-226a38c89794-plugins-config-volume\") pod \"26718843-e793-46f7-a49e-226a38c89794\" (UID: \"26718843-e793-46f7-a49e-226a38c89794\") " Nov 03 09:21:06.945426 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.945075 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26718843-e793-46f7-a49e-226a38c89794-kube-api-access-ssk2j\" (UniqueName: \"kubernetes.io/projected/26718843-e793-46f7-a49e-226a38c89794-kube-api-access-ssk2j\") pod \"26718843-e793-46f7-a49e-226a38c89794\" (UID: \"26718843-e793-46f7-a49e-226a38c89794\") " Nov 03 09:21:06.948542 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.948398 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26718843-e793-46f7-a49e-226a38c89794-plugins-config-volume" pod "26718843-e793-46f7-a49e-226a38c89794" (UID: "26718843-e793-46f7-a49e-226a38c89794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:21:06.948680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.948655 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26718843-e793-46f7-a49e-226a38c89794-kube-api-access-ssk2j" pod "26718843-e793-46f7-a49e-226a38c89794" (UID: "26718843-e793-46f7-a49e-226a38c89794"). InnerVolumeSpecName "kube-api-access-ssk2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:21:06.954714 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.954683 5015 generic.go:356] "Generic (PLEG): container finished" podID="26718843-e793-46f7-a49e-226a38c89794" containerID="1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758" exitCode=0 Nov 03 09:21:06.954872 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.954858 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" Nov 03 09:21:06.955462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.955239 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" event={"ID":"26718843-e793-46f7-a49e-226a38c89794","Type":"ContainerDied","Data":"1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758"} Nov 03 09:21:06.955462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.955308 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm" event={"ID":"26718843-e793-46f7-a49e-226a38c89794","Type":"ContainerDied","Data":"7d5cfc6992dc550871e82fdf432da856584d882ec9da44112d717030df4e6ed2"} Nov 03 09:21:06.955462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.955342 5015 scope.go:122] "RemoveContainer" containerID="1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758" Nov 03 09:21:06.957549 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.957526 5015 generic.go:356] "Generic (PLEG): container finished" podID="cc657df5-bac1-45a9-a46e-2ed9e2a99e79" containerID="6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3" exitCode=0 Nov 03 09:21:06.957650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.957574 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" Nov 03 09:21:06.957757 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.957736 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" event={"ID":"cc657df5-bac1-45a9-a46e-2ed9e2a99e79","Type":"ContainerDied","Data":"6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3"} Nov 03 09:21:06.957844 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.957789 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2" event={"ID":"cc657df5-bac1-45a9-a46e-2ed9e2a99e79","Type":"ContainerDied","Data":"1de28f3acbdcaa2c9b92b9cafb9169369b4547631d46e931e99e3427565d11a4"} Nov 03 09:21:06.975773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.975740 5015 scope.go:122] "RemoveContainer" containerID="1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758" Nov 03 09:21:06.976631 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:21:06.976593 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758\": container with ID starting with 1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758 not found: ID does not exist" containerID="1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758" Nov 03 09:21:06.976751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.976650 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758"} err="failed to get container status \"1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758\": rpc error: code = NotFound desc = could not find container \"1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758\": container with ID starting with 1fe70337db1eaede182b779118a46ba1469d36b325ea9300e1ba25c04240f758 not found: ID does not exist" Nov 03 09:21:06.976751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.976677 5015 scope.go:122] "RemoveContainer" containerID="6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3" Nov 03 09:21:06.988378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.988099 5015 scope.go:122] "RemoveContainer" containerID="6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3" Nov 03 09:21:06.988453 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:21:06.988376 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3\": container with ID starting with 6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3 not found: ID does not exist" containerID="6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3" Nov 03 09:21:06.988541 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:06.988408 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3"} err="failed to get container status \"6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3\": rpc error: code = NotFound desc = could not find container \"6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3\": container with ID starting with 6573bc549a44098ecd50c97c2efb446cd5d7c9fd3967bbee385dca43c30e81c3 not found: ID does not exist" Nov 03 09:21:07.046430 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.046353 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-kube-api-access-k756g\" (UniqueName: \"kubernetes.io/projected/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-kube-api-access-k756g\") pod \"cc657df5-bac1-45a9-a46e-2ed9e2a99e79\" (UID: \"cc657df5-bac1-45a9-a46e-2ed9e2a99e79\") " Nov 03 09:21:07.046585 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.046511 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-plugins-config-volume\") pod \"cc657df5-bac1-45a9-a46e-2ed9e2a99e79\" (UID: \"cc657df5-bac1-45a9-a46e-2ed9e2a99e79\") " Nov 03 09:21:07.046946 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.046909 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-plugins-config-volume" pod "cc657df5-bac1-45a9-a46e-2ed9e2a99e79" (UID: "cc657df5-bac1-45a9-a46e-2ed9e2a99e79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:21:07.047629 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.047603 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26718843-e793-46f7-a49e-226a38c89794-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:07.047729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.047634 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:07.047729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.047652 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssk2j\" (UniqueName: \"kubernetes.io/projected/26718843-e793-46f7-a49e-226a38c89794-kube-api-access-ssk2j\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:07.049159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.049133 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-kube-api-access-k756g" pod "cc657df5-bac1-45a9-a46e-2ed9e2a99e79" (UID: "cc657df5-bac1-45a9-a46e-2ed9e2a99e79"). InnerVolumeSpecName "kube-api-access-k756g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:21:07.149786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.149749 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k756g\" (UniqueName: \"kubernetes.io/projected/cc657df5-bac1-45a9-a46e-2ed9e2a99e79-kube-api-access-k756g\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:07.166741 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.166708 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm"] Nov 03 09:21:07.171132 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.171093 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6f8ed0-f8c7f4dc5-9bsbm"] Nov 03 09:21:07.207950 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.207915 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26718843-e793-46f7-a49e-226a38c89794" path="/var/lib/kubelet/pods/26718843-e793-46f7-a49e-226a38c89794/volumes" Nov 03 09:21:07.379104 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.379008 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2"] Nov 03 09:21:07.385254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:07.385226 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6f8ed0-588c7b8d99-2nfc2"] Nov 03 09:21:09.209039 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:09.209001 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc657df5-bac1-45a9-a46e-2ed9e2a99e79" path="/var/lib/kubelet/pods/cc657df5-bac1-45a9-a46e-2ed9e2a99e79/volumes" Nov 03 09:21:11.486900 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.486869 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7"] Nov 03 09:21:11.487534 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.487519 5015 cpu_manager.go:414] "containerMap: removing container" podUID="cc657df5-bac1-45a9-a46e-2ed9e2a99e79" containerName="payload-pre-processing" Nov 03 09:21:11.487534 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.487534 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc657df5-bac1-45a9-a46e-2ed9e2a99e79" containerName="payload-pre-processing" Nov 03 09:21:11.487655 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.487559 5015 cpu_manager.go:414] "containerMap: removing container" podUID="26718843-e793-46f7-a49e-226a38c89794" containerName="payload-processing" Nov 03 09:21:11.487655 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.487564 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26718843-e793-46f7-a49e-226a38c89794" containerName="payload-processing" Nov 03 09:21:11.487720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.487693 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc657df5-bac1-45a9-a46e-2ed9e2a99e79" containerName="payload-pre-processing" Nov 03 09:21:11.487720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.487705 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="26718843-e793-46f7-a49e-226a38c89794" containerName="payload-processing" Nov 03 09:21:11.492906 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.492887 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:21:11.550579 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.550546 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7"] Nov 03 09:21:11.603959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.603816 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-9ed110\"" Nov 03 09:21:11.604168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.603976 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-9ed110-dockercfg-bmhw4\"" Nov 03 09:21:11.655915 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.655871 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4x9f\" (UniqueName: \"kubernetes.io/projected/20ad04c6-7102-4548-b311-81690a614cc9-kube-api-access-q4x9f\") pod \"payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7\" (UID: \"20ad04c6-7102-4548-b311-81690a614cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:21:11.656166 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.656002 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ad04c6-7102-4548-b311-81690a614cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7\" (UID: \"20ad04c6-7102-4548-b311-81690a614cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:21:11.672845 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.672805 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8"] Nov 03 09:21:11.695528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.693272 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:21:11.719811 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.719776 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8"] Nov 03 09:21:11.757841 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.757760 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ad04c6-7102-4548-b311-81690a614cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7\" (UID: \"20ad04c6-7102-4548-b311-81690a614cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:21:11.758008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.757845 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8\" (UID: \"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:21:11.758077 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.758043 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h45pv\" (UniqueName: \"kubernetes.io/projected/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-kube-api-access-h45pv\") pod \"payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8\" (UID: \"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:21:11.758154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.758131 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4x9f\" (UniqueName: \"kubernetes.io/projected/20ad04c6-7102-4548-b311-81690a614cc9-kube-api-access-q4x9f\") pod \"payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7\" (UID: \"20ad04c6-7102-4548-b311-81690a614cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:21:11.758667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.758642 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ad04c6-7102-4548-b311-81690a614cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7\" (UID: \"20ad04c6-7102-4548-b311-81690a614cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:21:11.817506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.817453 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4x9f\" (UniqueName: \"kubernetes.io/projected/20ad04c6-7102-4548-b311-81690a614cc9-kube-api-access-q4x9f\") pod \"payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7\" (UID: \"20ad04c6-7102-4548-b311-81690a614cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:21:11.859495 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.859443 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8\" (UID: \"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:21:11.859700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.859598 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h45pv\" (UniqueName: \"kubernetes.io/projected/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-kube-api-access-h45pv\") pod \"payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8\" (UID: \"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:21:11.860133 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.860109 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8\" (UID: \"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:21:11.919118 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:11.919080 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h45pv\" (UniqueName: \"kubernetes.io/projected/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-kube-api-access-h45pv\") pod \"payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8\" (UID: \"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:21:12.034064 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:12.033981 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:21:12.119607 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:12.117785 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:21:12.421903 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:12.421870 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8"] Nov 03 09:21:12.422146 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:21:12.422127 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa279c1_6afa_4a7e_b57c_8f3fbd95cfdb.slice/crio-31a3e5982d41f4b68b584b3506a50fb907d4c317ccbead160a1cf76c0f96d7b5 WatchSource:0}: Error finding container 31a3e5982d41f4b68b584b3506a50fb907d4c317ccbead160a1cf76c0f96d7b5: Status 404 returned error can't find the container with id 31a3e5982d41f4b68b584b3506a50fb907d4c317ccbead160a1cf76c0f96d7b5 Nov 03 09:21:12.528367 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:12.528332 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7"] Nov 03 09:21:12.528850 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:21:12.528821 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ad04c6_7102_4548_b311_81690a614cc9.slice/crio-8ff04633091591c9e825363efb6bb36cbefaeec27d81458e33579386b38e70a0 WatchSource:0}: Error finding container 8ff04633091591c9e825363efb6bb36cbefaeec27d81458e33579386b38e70a0: Status 404 returned error can't find the container with id 8ff04633091591c9e825363efb6bb36cbefaeec27d81458e33579386b38e70a0 Nov 03 09:21:13.013986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:13.013936 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" event={"ID":"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb","Type":"ContainerStarted","Data":"8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db"} Nov 03 09:21:13.014197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:13.013996 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" event={"ID":"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb","Type":"ContainerStarted","Data":"31a3e5982d41f4b68b584b3506a50fb907d4c317ccbead160a1cf76c0f96d7b5"} Nov 03 09:21:13.014197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:13.014023 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:21:13.015746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:13.015720 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" event={"ID":"20ad04c6-7102-4548-b311-81690a614cc9","Type":"ContainerStarted","Data":"8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072"} Nov 03 09:21:13.015862 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:13.015769 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:21:13.015862 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:13.015784 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" event={"ID":"20ad04c6-7102-4548-b311-81690a614cc9","Type":"ContainerStarted","Data":"8ff04633091591c9e825363efb6bb36cbefaeec27d81458e33579386b38e70a0"} Nov 03 09:21:13.131440 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:13.131383 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" podStartSLOduration=2.131367503 podStartE2EDuration="2.131367503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:21:13.129996483 +0000 UTC m=+2506.054938691" watchObservedRunningTime="2026-11-03 09:21:13.131367503 +0000 UTC m=+2506.056309720" Nov 03 09:21:13.260020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:13.259955 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" podStartSLOduration=2.259932435 podStartE2EDuration="2.259932435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:21:13.254835559 +0000 UTC m=+2506.179777778" watchObservedRunningTime="2026-11-03 09:21:13.259932435 +0000 UTC m=+2506.184874656" Nov 03 09:21:15.594509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.594438 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2"] Nov 03 09:21:15.599261 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.599239 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:15.658982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.658947 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2"] Nov 03 09:21:15.702593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.702558 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-b196c8-serving-cert\"" Nov 03 09:21:15.712854 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.712820 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:15.713025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.712864 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:15.713025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.712927 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdlx\" (UniqueName: \"kubernetes.io/projected/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-kube-api-access-2vdlx\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:15.713166 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.713139 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:15.814200 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.814141 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:15.814401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.814218 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:15.814401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.814250 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:15.814401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.814287 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdlx\" (UniqueName: \"kubernetes.io/projected/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-kube-api-access-2vdlx\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:15.814401 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:21:15.814323 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-serving-cert: secret "maas-api-e2e-shared-b-w3-b196c8-serving-cert" not found Nov 03 09:21:15.814721 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:21:15.814442 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-maas-api-tls podName:190ecc81-1e36-4a1d-9b19-d1f9f2d06b15 nodeName:}" failed. No retries permitted until 2026-11-03 09:21:16.314419429 +0000 UTC m=+2509.239361624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-maas-api-tls") pod "maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" (UID: "190ecc81-1e36-4a1d-9b19-d1f9f2d06b15") : secret "maas-api-e2e-shared-b-w3-b196c8-serving-cert" not found Nov 03 09:21:15.814983 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.814965 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:15.816965 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.816946 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:15.875457 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:15.875382 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdlx\" (UniqueName: \"kubernetes.io/projected/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-kube-api-access-2vdlx\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:16.323120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:16.323082 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:16.325785 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:16.325759 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:16.514716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:16.514678 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:16.906244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:16.906206 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2"] Nov 03 09:21:16.908881 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:21:16.908849 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190ecc81_1e36_4a1d_9b19_d1f9f2d06b15.slice/crio-10c13d7a026351fe8c235aa43868ef3c41663a52c545e23db96a82ae40dac03b WatchSource:0}: Error finding container 10c13d7a026351fe8c235aa43868ef3c41663a52c545e23db96a82ae40dac03b: Status 404 returned error can't find the container with id 10c13d7a026351fe8c235aa43868ef3c41663a52c545e23db96a82ae40dac03b Nov 03 09:21:16.912698 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:21:16.912654 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:21:17.053211 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:17.053172 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" event={"ID":"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15","Type":"ContainerStarted","Data":"10c13d7a026351fe8c235aa43868ef3c41663a52c545e23db96a82ae40dac03b"} Nov 03 09:21:18.075586 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:18.075442 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" event={"ID":"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15","Type":"ContainerStarted","Data":"a8eec25e0cd1079846ba08119a4d46ae8fe6c91f25069751573fa41db93b6a9b"} Nov 03 09:21:18.076066 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:18.075589 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:18.193782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:18.193716 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" podStartSLOduration=2.309553097 podStartE2EDuration="3.193699178s" totalImagesPullingTime="884.146081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:21:18.192993427 +0000 UTC m=+2511.117935656" watchObservedRunningTime="2026-11-03 09:21:18.193699178 +0000 UTC m=+2511.118641396" Nov 03 09:21:24.028214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:24.028185 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:21:24.028657 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:24.028251 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:21:24.092281 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:21:24.092251 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:21:46.904129 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:21:46.904086 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:22:16.903650 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:22:16.903570 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:22:20.011138 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.011097 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz"] Nov 03 09:22:20.017610 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.017585 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:22:20.036601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.036570 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz"] Nov 03 09:22:20.079956 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.079850 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-2fc57e-dockercfg-wvg95\"" Nov 03 09:22:20.080272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.080254 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-2fc57e\"" Nov 03 09:22:20.098329 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.098293 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777f762c-f419-479f-a1d1-8cb613249b1e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz\" (UID: \"777f762c-f419-479f-a1d1-8cb613249b1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:22:20.098553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.098408 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlh25\" (UniqueName: \"kubernetes.io/projected/777f762c-f419-479f-a1d1-8cb613249b1e-kube-api-access-dlh25\") pod \"payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz\" (UID: \"777f762c-f419-479f-a1d1-8cb613249b1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:22:20.199779 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.199743 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777f762c-f419-479f-a1d1-8cb613249b1e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz\" (UID: \"777f762c-f419-479f-a1d1-8cb613249b1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:22:20.199953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.199798 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlh25\" (UniqueName: \"kubernetes.io/projected/777f762c-f419-479f-a1d1-8cb613249b1e-kube-api-access-dlh25\") pod \"payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz\" (UID: \"777f762c-f419-479f-a1d1-8cb613249b1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:22:20.200599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.200579 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777f762c-f419-479f-a1d1-8cb613249b1e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz\" (UID: \"777f762c-f419-479f-a1d1-8cb613249b1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:22:20.259611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.259577 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlh25\" (UniqueName: \"kubernetes.io/projected/777f762c-f419-479f-a1d1-8cb613249b1e-kube-api-access-dlh25\") pod \"payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz\" (UID: \"777f762c-f419-479f-a1d1-8cb613249b1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:22:20.341746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.341666 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:22:20.731007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:20.730974 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz"] Nov 03 09:22:20.732274 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:22:20.732249 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777f762c_f419_479f_a1d1_8cb613249b1e.slice/crio-7d5a4ec2938aea702a40cc74455de9c49ade4082211f0a95354074102d4e0b02 WatchSource:0}: Error finding container 7d5a4ec2938aea702a40cc74455de9c49ade4082211f0a95354074102d4e0b02: Status 404 returned error can't find the container with id 7d5a4ec2938aea702a40cc74455de9c49ade4082211f0a95354074102d4e0b02 Nov 03 09:22:21.667199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:21.667153 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" event={"ID":"777f762c-f419-479f-a1d1-8cb613249b1e","Type":"ContainerStarted","Data":"00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb"} Nov 03 09:22:21.667729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:21.667255 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" event={"ID":"777f762c-f419-479f-a1d1-8cb613249b1e","Type":"ContainerStarted","Data":"7d5a4ec2938aea702a40cc74455de9c49ade4082211f0a95354074102d4e0b02"} Nov 03 09:22:21.667729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:21.667298 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:22:21.786838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:21.786770 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" podStartSLOduration=2.78675086 podStartE2EDuration="2.78675086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:22:21.784272272 +0000 UTC m=+2574.709214512" watchObservedRunningTime="2026-11-03 09:22:21.78675086 +0000 UTC m=+2574.711693078" Nov 03 09:22:32.678337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:32.678305 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:22:46.904235 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:22:46.904180 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:22:56.772050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:56.772009 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk"] Nov 03 09:22:56.773093 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:22:56.773035 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerName="istio-proxy" containerID="cri-o://34e9c41a504757a171ffe186e72ebaa6a555725dcd3862b97ccb8c446640e506" gracePeriod=30 Nov 03 09:23:01.830470 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:01.830435 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 09:23:01.830856 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:01.830524 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 09:23:02.042493 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.042396 5015 generic.go:356] "Generic (PLEG): container finished" podID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerID="34e9c41a504757a171ffe186e72ebaa6a555725dcd3862b97ccb8c446640e506" exitCode=0 Nov 03 09:23:02.042638 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.042510 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" event={"ID":"2e72455e-e27f-41c7-99e0-6dcabd1a2882","Type":"ContainerDied","Data":"34e9c41a504757a171ffe186e72ebaa6a555725dcd3862b97ccb8c446640e506"} Nov 03 09:23:02.042638 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.042580 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" event={"ID":"2e72455e-e27f-41c7-99e0-6dcabd1a2882","Type":"ContainerDied","Data":"2fe9e77912bf1ff9bbb5c051244fa3652e4faac5296bb969d6adedf966968433"} Nov 03 09:23:02.042638 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.042605 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fe9e77912bf1ff9bbb5c051244fa3652e4faac5296bb969d6adedf966968433" Nov 03 09:23:02.063196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.063169 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:23:02.181100 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.181066 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-socket\") pod \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " Nov 03 09:23:02.181407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.181113 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istiod-ca-cert\") pod \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " Nov 03 09:23:02.181407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.181151 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-podinfo\") pod \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " Nov 03 09:23:02.181407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.181218 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-kube-api-access-2xgkq\" (UniqueName: \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-kube-api-access-2xgkq\") pod \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " Nov 03 09:23:02.181407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.181238 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-certs\") pod \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " Nov 03 09:23:02.181407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.181288 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-envoy\") pod \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " Nov 03 09:23:02.181407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.181372 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-token\" (UniqueName: \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-token\") pod \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " Nov 03 09:23:02.181755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.181440 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-data\") pod \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " Nov 03 09:23:02.181755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.181465 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-credential-socket\") pod \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\" (UID: \"2e72455e-e27f-41c7-99e0-6dcabd1a2882\") " Nov 03 09:23:02.182442 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.182398 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-data" pod "2e72455e-e27f-41c7-99e0-6dcabd1a2882" (UID: "2e72455e-e27f-41c7-99e0-6dcabd1a2882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.182442 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.182412 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istiod-ca-cert" pod "2e72455e-e27f-41c7-99e0-6dcabd1a2882" (UID: "2e72455e-e27f-41c7-99e0-6dcabd1a2882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:23:02.182720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.182670 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-certs" pod "2e72455e-e27f-41c7-99e0-6dcabd1a2882" (UID: "2e72455e-e27f-41c7-99e0-6dcabd1a2882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.182958 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.182932 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-credential-socket" pod "2e72455e-e27f-41c7-99e0-6dcabd1a2882" (UID: "2e72455e-e27f-41c7-99e0-6dcabd1a2882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.183073 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.183029 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-socket" pod "2e72455e-e27f-41c7-99e0-6dcabd1a2882" (UID: "2e72455e-e27f-41c7-99e0-6dcabd1a2882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.184824 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.184798 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-podinfo" pod "2e72455e-e27f-41c7-99e0-6dcabd1a2882" (UID: "2e72455e-e27f-41c7-99e0-6dcabd1a2882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:23:02.185113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.185088 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-token" pod "2e72455e-e27f-41c7-99e0-6dcabd1a2882" (UID: "2e72455e-e27f-41c7-99e0-6dcabd1a2882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:23:02.185289 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.185262 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-envoy" pod "2e72455e-e27f-41c7-99e0-6dcabd1a2882" (UID: "2e72455e-e27f-41c7-99e0-6dcabd1a2882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.185380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.185306 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-kube-api-access-2xgkq" pod "2e72455e-e27f-41c7-99e0-6dcabd1a2882" (UID: "2e72455e-e27f-41c7-99e0-6dcabd1a2882"). InnerVolumeSpecName "kube-api-access-2xgkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:23:02.283966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.283926 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xgkq\" (UniqueName: \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-kube-api-access-2xgkq\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.283966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.283966 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.283966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.283981 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.283966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.283998 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.284389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.284014 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.284389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.284028 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.284389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.284041 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e72455e-e27f-41c7-99e0-6dcabd1a2882-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.284389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.284055 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.284389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:02.284068 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e72455e-e27f-41c7-99e0-6dcabd1a2882-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:03.051502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:03.051441 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk" Nov 03 09:23:03.276562 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:03.276426 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk"] Nov 03 09:23:03.280792 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:03.280767 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-mn5tk"] Nov 03 09:23:04.068834 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:04.068795 5015 generic.go:356] "Generic (PLEG): container finished" podID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerID="77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb" exitCode=0 Nov 03 09:23:04.069262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:04.068854 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" event={"ID":"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f","Type":"ContainerDied","Data":"77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb"} Nov 03 09:23:04.069402 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:04.069385 5015 scope.go:122] "RemoveContainer" containerID="77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb" Nov 03 09:23:04.071330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:04.071296 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:23:04.071449 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:04.071390 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:23:04.071546 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:04.071448 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:23:05.079792 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:05.079750 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" event={"ID":"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f","Type":"ContainerStarted","Data":"4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb"} Nov 03 09:23:05.205687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:05.205644 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" path="/var/lib/kubelet/pods/2e72455e-e27f-41c7-99e0-6dcabd1a2882/volumes" Nov 03 09:23:05.265509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:05.264711 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:23:05.270305 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:05.270266 5015 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 09:23:05.270531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:05.270332 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 09:23:05.963502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:05.944576 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:23:06.260573 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:06.259873 5015 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 09:23:06.260573 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:06.259934 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 09:23:07.268807 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:07.259821 5015 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 09:23:07.268807 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:07.259883 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 09:23:08.260332 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:08.260296 5015 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 09:23:08.260660 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:08.260633 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 09:23:09.259617 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:09.259577 5015 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 09:23:09.260217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:09.260187 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 09:23:09.964076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:09.964027 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv"] Nov 03 09:23:09.966251 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:09.966229 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerName="istio-proxy" Nov 03 09:23:09.966397 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:09.966384 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerName="istio-proxy" Nov 03 09:23:09.974389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:09.973970 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e72455e-e27f-41c7-99e0-6dcabd1a2882" containerName="istio-proxy" Nov 03 09:23:09.978702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:09.978680 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.009693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.009665 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv"] Nov 03 09:23:10.032252 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.032215 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gddxf\" (UniqueName: \"kubernetes.io/projected/d1fb16c2-c2aa-474a-9567-b85f640c4110-kube-api-access-gddxf\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.032414 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.032340 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb16c2-c2aa-474a-9567-b85f640c4110-tls-certs\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.032506 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.032432 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-home\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.032582 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.032563 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-tmp-dir\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.032640 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.032595 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-dshm\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.032698 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.032685 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-model-cache\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.071501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.061915 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-627e5f\"/\"kube-root-ca.crt\"" Nov 03 09:23:10.071501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.062069 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-627e5f\"/\"default-dockercfg-jb48d\"" Nov 03 09:23:10.071501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.062135 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-627e5f\"/\"openshift-service-ca.crt\"" Nov 03 09:23:10.082469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.082399 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-627e5f\"/\"auth-test-model-627e5f-kserve-self-signed-certs\"" Nov 03 09:23:10.134112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.134075 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-tmp-dir\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.134112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.134116 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-dshm\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.134393 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.134180 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-model-cache\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.134393 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.134225 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gddxf\" (UniqueName: \"kubernetes.io/projected/d1fb16c2-c2aa-474a-9567-b85f640c4110-kube-api-access-gddxf\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.134393 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.134307 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb16c2-c2aa-474a-9567-b85f640c4110-tls-certs\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.134393 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.134345 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-home\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.134670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.134580 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-tmp-dir\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.134708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.134677 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-model-cache\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.134780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.134760 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-home\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.161570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.136668 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-dshm\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.161570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.160761 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb16c2-c2aa-474a-9567-b85f640c4110-tls-certs\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.193874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.193844 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gddxf\" (UniqueName: \"kubernetes.io/projected/d1fb16c2-c2aa-474a-9567-b85f640c4110-kube-api-access-gddxf\") pod \"auth-test-model-627e5f-kserve-854b76dc89-dmsxv\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.260514 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.259856 5015 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 09:23:10.260514 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.259933 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 09:23:10.294363 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.294328 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:10.700945 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:23:10.700915 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1fb16c2_c2aa_474a_9567_b85f640c4110.slice/crio-429da9d8a3b36c9dabf3b0e58f61b2afa49f30477be88c3ddf7dcf4cf1c33aab WatchSource:0}: Error finding container 429da9d8a3b36c9dabf3b0e58f61b2afa49f30477be88c3ddf7dcf4cf1c33aab: Status 404 returned error can't find the container with id 429da9d8a3b36c9dabf3b0e58f61b2afa49f30477be88c3ddf7dcf4cf1c33aab Nov 03 09:23:10.700945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:10.700919 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv"] Nov 03 09:23:11.127811 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:11.127770 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" event={"ID":"d1fb16c2-c2aa-474a-9567-b85f640c4110","Type":"ContainerStarted","Data":"06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b"} Nov 03 09:23:11.128000 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:11.127820 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" event={"ID":"d1fb16c2-c2aa-474a-9567-b85f640c4110","Type":"ContainerStarted","Data":"429da9d8a3b36c9dabf3b0e58f61b2afa49f30477be88c3ddf7dcf4cf1c33aab"} Nov 03 09:23:11.247290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:11.247227 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" podStartSLOduration=2.247213534 podStartE2EDuration="2.247213534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:23:11.243622562 +0000 UTC m=+2624.168564780" watchObservedRunningTime="2026-11-03 09:23:11.247213534 +0000 UTC m=+2624.172155801" Nov 03 09:23:11.264551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:11.264523 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:23:11.265767 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:11.265744 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:23:11.294876 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:11.294850 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:16.903407 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:23:16.903367 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:23:20.294802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:20.294770 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:20.307908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:20.307879 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:21.218849 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:21.218822 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:23:28.382925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:28.382890 5015 scope.go:122] "RemoveContainer" containerID="34e9c41a504757a171ffe186e72ebaa6a555725dcd3862b97ccb8c446640e506" Nov 03 09:23:46.903848 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:23:46.903765 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:23:59.976785 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:59.976754 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2"] Nov 03 09:23:59.981440 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:59.981418 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:23:59.991400 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:23:59.991376 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2"] Nov 03 09:24:00.038811 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.038778 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-506e6d\"/\"openshift-service-ca.crt\"" Nov 03 09:24:00.039057 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.039038 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-506e6d\"/\"kube-root-ca.crt\"" Nov 03 09:24:00.039057 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.039048 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-506e6d\"/\"auth-test-model-506e6d-kserve-self-signed-certs\"" Nov 03 09:24:00.039194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.039107 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-506e6d\"/\"default-dockercfg-s654v\"" Nov 03 09:24:00.054379 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.054338 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89525c49-48f6-4822-9a33-159ede94f8af-tls-certs\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.054840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.054782 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-dshm\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.055108 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.055081 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-tmp-dir\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.055983 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.055443 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-home\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.055983 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.055643 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-model-cache\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.055983 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.055730 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dmzs\" (UniqueName: \"kubernetes.io/projected/89525c49-48f6-4822-9a33-159ede94f8af-kube-api-access-8dmzs\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.157179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.157149 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-home\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.157362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.157251 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-model-cache\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.157362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.157286 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dmzs\" (UniqueName: \"kubernetes.io/projected/89525c49-48f6-4822-9a33-159ede94f8af-kube-api-access-8dmzs\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.157362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.157347 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89525c49-48f6-4822-9a33-159ede94f8af-tls-certs\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.157517 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.157443 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-dshm\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.157647 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.157625 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-tmp-dir\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.157731 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.157668 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-home\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.157731 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.157695 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-model-cache\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.157928 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.157910 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-tmp-dir\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.160134 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.160102 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-dshm\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.160294 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.160277 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89525c49-48f6-4822-9a33-159ede94f8af-tls-certs\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.216665 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.216634 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dmzs\" (UniqueName: \"kubernetes.io/projected/89525c49-48f6-4822-9a33-159ede94f8af-kube-api-access-8dmzs\") pod \"auth-test-model-506e6d-kserve-cf9c44645-rhjf2\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.295974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.295878 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:00.713136 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:24:00.713103 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89525c49_48f6_4822_9a33_159ede94f8af.slice/crio-edb6dab8c6a75523258703836d3c2443fff27b85902ae420e91551d092acf05f WatchSource:0}: Error finding container edb6dab8c6a75523258703836d3c2443fff27b85902ae420e91551d092acf05f: Status 404 returned error can't find the container with id edb6dab8c6a75523258703836d3c2443fff27b85902ae420e91551d092acf05f Nov 03 09:24:00.713897 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:00.713867 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2"] Nov 03 09:24:01.556034 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:01.555994 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" event={"ID":"89525c49-48f6-4822-9a33-159ede94f8af","Type":"ContainerStarted","Data":"2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354"} Nov 03 09:24:01.556613 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:01.556054 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" event={"ID":"89525c49-48f6-4822-9a33-159ede94f8af","Type":"ContainerStarted","Data":"edb6dab8c6a75523258703836d3c2443fff27b85902ae420e91551d092acf05f"} Nov 03 09:24:01.674786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:01.674723 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" podStartSLOduration=2.67470976 podStartE2EDuration="2.67470976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:24:01.672801453 +0000 UTC m=+2674.597743701" watchObservedRunningTime="2026-11-03 09:24:01.67470976 +0000 UTC m=+2674.599651977" Nov 03 09:24:02.296549 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:02.296515 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:02.672887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:02.672795 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw"] Nov 03 09:24:02.673297 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:02.673004 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerName="istio-proxy" containerID="cri-o://291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8" gracePeriod=30 Nov 03 09:24:07.960531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.960444 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:24:07.977590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.977561 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-podinfo\") pod \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " Nov 03 09:24:07.977753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.977608 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-kube-api-access-rnqhp\" (UniqueName: \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-kube-api-access-rnqhp\") pod \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " Nov 03 09:24:07.977753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.977643 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-data\") pod \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " Nov 03 09:24:07.977753 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.977690 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-token\" (UniqueName: \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-token\") pod \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " Nov 03 09:24:07.977932 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.977816 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-socket\") pod \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " Nov 03 09:24:07.977996 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.977942 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-credential-socket\") pod \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " Nov 03 09:24:07.977996 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.977949 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-data" pod "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" (UID: "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:24:07.978109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978039 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istiod-ca-cert\") pod \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " Nov 03 09:24:07.978109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978071 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-certs\") pod \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " Nov 03 09:24:07.978109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978094 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-envoy\") pod \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\" (UID: \"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa\") " Nov 03 09:24:07.978270 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978116 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-socket" pod "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" (UID: "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:24:07.978333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978284 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-credential-socket" pod "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" (UID: "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:24:07.978405 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978340 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-certs" pod "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" (UID: "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:24:07.978405 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978384 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istiod-ca-cert" pod "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" (UID: "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:24:07.978713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978681 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:24:07.978713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978703 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:24:07.978713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978717 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:24:07.978713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978730 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:24:07.978713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.978746 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:24:07.980304 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.980273 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-token" pod "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" (UID: "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:24:07.980425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.980360 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-podinfo" pod "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" (UID: "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:24:07.980903 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.980876 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-kube-api-access-rnqhp" pod "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" (UID: "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa"). InnerVolumeSpecName "kube-api-access-rnqhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:24:07.980982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:07.980965 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-envoy" pod "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" (UID: "57e7beb7-7cc4-4dd7-8112-9c099f6afcaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:24:08.080514 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.080465 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:24:08.080514 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.080512 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnqhp\" (UniqueName: \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-kube-api-access-rnqhp\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:24:08.080514 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.080523 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:24:08.080758 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.080532 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:24:08.621317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.621280 5015 generic.go:356] "Generic (PLEG): container finished" podID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerID="291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8" exitCode=0 Nov 03 09:24:08.621574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.621341 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" Nov 03 09:24:08.621574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.621376 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" event={"ID":"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa","Type":"ContainerDied","Data":"291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8"} Nov 03 09:24:08.621574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.621438 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw" event={"ID":"57e7beb7-7cc4-4dd7-8112-9c099f6afcaa","Type":"ContainerDied","Data":"c52a716360b53e8a7db727f83c68d3b633ab193a5cf857ad3c727a6de9631863"} Nov 03 09:24:08.621574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.621467 5015 scope.go:122] "RemoveContainer" containerID="291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8" Nov 03 09:24:08.639077 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.639041 5015 scope.go:122] "RemoveContainer" containerID="291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8" Nov 03 09:24:08.639412 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:24:08.639388 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8\": container with ID starting with 291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8 not found: ID does not exist" containerID="291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8" Nov 03 09:24:08.639512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.639420 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8"} err="failed to get container status \"291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8\": rpc error: code = NotFound desc = could not find container \"291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8\": container with ID starting with 291640be2dd430a0e4ce2d6fe40688c05d8aa792f583fce3d9b2d558485e57f8 not found: ID does not exist" Nov 03 09:24:08.794193 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.794162 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw"] Nov 03 09:24:08.844641 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:08.844608 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-7glnw"] Nov 03 09:24:09.205873 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:09.205842 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" path="/var/lib/kubelet/pods/57e7beb7-7cc4-4dd7-8112-9c099f6afcaa/volumes" Nov 03 09:24:10.296109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:10.296079 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:10.316797 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:10.316770 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:10.654632 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:10.654546 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:24:16.903696 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:24:16.903657 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:24:27.705217 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:27.705185 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:24:27.719164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:24:27.719141 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:24:46.905287 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:24:46.905245 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:25:16.903680 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:25:16.903588 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:25:43.062411 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:43.062369 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96"] Nov 03 09:25:43.062890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:43.062605 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerName="istio-proxy" containerID="cri-o://81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592" gracePeriod=30 Nov 03 09:25:46.904008 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:25:46.903966 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:25:46.978191 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:46.978159 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s"] Nov 03 09:25:46.978378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:46.978356 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerName="istio-proxy" containerID="cri-o://5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe" gracePeriod=30 Nov 03 09:25:48.348011 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.347939 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:25:48.380112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380079 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-certs\") pod \"5d233c76-e29e-43a0-abd0-212ce5df00a3\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " Nov 03 09:25:48.380112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380126 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-data\") pod \"5d233c76-e29e-43a0-abd0-212ce5df00a3\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " Nov 03 09:25:48.380388 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380194 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-envoy\") pod \"5d233c76-e29e-43a0-abd0-212ce5df00a3\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " Nov 03 09:25:48.380388 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380242 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-token\" (UniqueName: \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-token\") pod \"5d233c76-e29e-43a0-abd0-212ce5df00a3\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " Nov 03 09:25:48.380388 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380369 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d233c76-e29e-43a0-abd0-212ce5df00a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d233c76-e29e-43a0-abd0-212ce5df00a3-istiod-ca-cert\") pod \"5d233c76-e29e-43a0-abd0-212ce5df00a3\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " Nov 03 09:25:48.380594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380400 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-certs" pod "5d233c76-e29e-43a0-abd0-212ce5df00a3" (UID: "5d233c76-e29e-43a0-abd0-212ce5df00a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:48.380594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380412 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-credential-socket\") pod \"5d233c76-e29e-43a0-abd0-212ce5df00a3\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " Nov 03 09:25:48.380594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380420 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-data" pod "5d233c76-e29e-43a0-abd0-212ce5df00a3" (UID: "5d233c76-e29e-43a0-abd0-212ce5df00a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:48.380594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380466 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-podinfo\") pod \"5d233c76-e29e-43a0-abd0-212ce5df00a3\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " Nov 03 09:25:48.380594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380544 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-kube-api-access-nmmvn\" (UniqueName: \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-kube-api-access-nmmvn\") pod \"5d233c76-e29e-43a0-abd0-212ce5df00a3\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " Nov 03 09:25:48.380860 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380612 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-socket\") pod \"5d233c76-e29e-43a0-abd0-212ce5df00a3\" (UID: \"5d233c76-e29e-43a0-abd0-212ce5df00a3\") " Nov 03 09:25:48.380860 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.380725 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-credential-socket" pod "5d233c76-e29e-43a0-abd0-212ce5df00a3" (UID: "5d233c76-e29e-43a0-abd0-212ce5df00a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:48.381136 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.381115 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:48.381198 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.381143 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:48.381198 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.381159 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:48.381308 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.381289 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d233c76-e29e-43a0-abd0-212ce5df00a3-istiod-ca-cert" pod "5d233c76-e29e-43a0-abd0-212ce5df00a3" (UID: "5d233c76-e29e-43a0-abd0-212ce5df00a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:25:48.381365 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.381346 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-socket" pod "5d233c76-e29e-43a0-abd0-212ce5df00a3" (UID: "5d233c76-e29e-43a0-abd0-212ce5df00a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:48.383199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.383159 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-envoy" pod "5d233c76-e29e-43a0-abd0-212ce5df00a3" (UID: "5d233c76-e29e-43a0-abd0-212ce5df00a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:48.383449 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.383424 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-token" pod "5d233c76-e29e-43a0-abd0-212ce5df00a3" (UID: "5d233c76-e29e-43a0-abd0-212ce5df00a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:25:48.383605 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.383581 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-kube-api-access-nmmvn" pod "5d233c76-e29e-43a0-abd0-212ce5df00a3" (UID: "5d233c76-e29e-43a0-abd0-212ce5df00a3"). InnerVolumeSpecName "kube-api-access-nmmvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:25:48.383672 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.383612 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-podinfo" pod "5d233c76-e29e-43a0-abd0-212ce5df00a3" (UID: "5d233c76-e29e-43a0-abd0-212ce5df00a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:25:48.481940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.481899 5015 generic.go:356] "Generic (PLEG): container finished" podID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerID="81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592" exitCode=0 Nov 03 09:25:48.482132 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.481959 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" Nov 03 09:25:48.482132 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.481963 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" event={"ID":"5d233c76-e29e-43a0-abd0-212ce5df00a3","Type":"ContainerDied","Data":"81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592"} Nov 03 09:25:48.482132 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.482030 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" event={"ID":"5d233c76-e29e-43a0-abd0-212ce5df00a3","Type":"ContainerDied","Data":"d732f0405c1699c78773fb5356e4ba458670c700e308730bea8605a2e7e50377"} Nov 03 09:25:48.482132 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.482053 5015 scope.go:122] "RemoveContainer" containerID="81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592" Nov 03 09:25:48.482360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.482324 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmmvn\" (UniqueName: \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-kube-api-access-nmmvn\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:48.482360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.482348 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:48.482499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.482365 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:48.482499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.482384 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:48.482499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.482400 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d233c76-e29e-43a0-abd0-212ce5df00a3-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:48.482499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.482412 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d233c76-e29e-43a0-abd0-212ce5df00a3-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:48.495989 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.495970 5015 scope.go:122] "RemoveContainer" containerID="81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592" Nov 03 09:25:48.496275 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:25:48.496255 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592\": container with ID starting with 81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592 not found: ID does not exist" containerID="81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592" Nov 03 09:25:48.496347 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.496289 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592"} err="failed to get container status \"81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592\": rpc error: code = NotFound desc = could not find container \"81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592\": container with ID starting with 81db1f749db7c1938a8f8aa7f26861268ffb15782e6a61eec07f1a93041fb592 not found: ID does not exist" Nov 03 09:25:48.636826 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.636741 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96"] Nov 03 09:25:48.643606 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:48.643579 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96"] Nov 03 09:25:49.206624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:49.206589 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" path="/var/lib/kubelet/pods/5d233c76-e29e-43a0-abd0-212ce5df00a3/volumes" Nov 03 09:25:49.298114 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:49.298072 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:25:49.298301 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:49.298147 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-9jt96" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:25:52.265655 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.265581 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 09:25:52.332162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332130 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-envoy\") pod \"b243ef81-e040-408a-96e6-6ce5c7df25af\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " Nov 03 09:25:52.332338 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332179 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-data\") pod \"b243ef81-e040-408a-96e6-6ce5c7df25af\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " Nov 03 09:25:52.332338 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332278 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-kube-api-access-kn9g7\" (UniqueName: \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-kube-api-access-kn9g7\") pod \"b243ef81-e040-408a-96e6-6ce5c7df25af\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " Nov 03 09:25:52.332338 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332331 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-socket\") pod \"b243ef81-e040-408a-96e6-6ce5c7df25af\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " Nov 03 09:25:52.332538 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332405 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-certs\") pod \"b243ef81-e040-408a-96e6-6ce5c7df25af\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " Nov 03 09:25:52.332538 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332510 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-data" pod "b243ef81-e040-408a-96e6-6ce5c7df25af" (UID: "b243ef81-e040-408a-96e6-6ce5c7df25af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:52.332663 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332543 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b243ef81-e040-408a-96e6-6ce5c7df25af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b243ef81-e040-408a-96e6-6ce5c7df25af-istiod-ca-cert\") pod \"b243ef81-e040-408a-96e6-6ce5c7df25af\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " Nov 03 09:25:52.332723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332659 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-certs" pod "b243ef81-e040-408a-96e6-6ce5c7df25af" (UID: "b243ef81-e040-408a-96e6-6ce5c7df25af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:52.332723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332678 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-socket" pod "b243ef81-e040-408a-96e6-6ce5c7df25af" (UID: "b243ef81-e040-408a-96e6-6ce5c7df25af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:52.332818 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332720 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-podinfo\") pod \"b243ef81-e040-408a-96e6-6ce5c7df25af\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " Nov 03 09:25:52.332818 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332759 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-token\" (UniqueName: \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-token\") pod \"b243ef81-e040-408a-96e6-6ce5c7df25af\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " Nov 03 09:25:52.332818 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332800 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-credential-socket\") pod \"b243ef81-e040-408a-96e6-6ce5c7df25af\" (UID: \"b243ef81-e040-408a-96e6-6ce5c7df25af\") " Nov 03 09:25:52.332919 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.332907 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b243ef81-e040-408a-96e6-6ce5c7df25af-istiod-ca-cert" pod "b243ef81-e040-408a-96e6-6ce5c7df25af" (UID: "b243ef81-e040-408a-96e6-6ce5c7df25af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:25:52.333248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.333228 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-credential-socket" pod "b243ef81-e040-408a-96e6-6ce5c7df25af" (UID: "b243ef81-e040-408a-96e6-6ce5c7df25af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:52.333422 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.333404 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:52.333708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.333424 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:52.333708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.333434 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:52.333708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.333443 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:52.333708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.333451 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b243ef81-e040-408a-96e6-6ce5c7df25af-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:52.334787 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.334759 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-kube-api-access-kn9g7" pod "b243ef81-e040-408a-96e6-6ce5c7df25af" (UID: "b243ef81-e040-408a-96e6-6ce5c7df25af"). InnerVolumeSpecName "kube-api-access-kn9g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:25:52.335304 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.335279 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-envoy" pod "b243ef81-e040-408a-96e6-6ce5c7df25af" (UID: "b243ef81-e040-408a-96e6-6ce5c7df25af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:52.335427 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.335321 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-podinfo" pod "b243ef81-e040-408a-96e6-6ce5c7df25af" (UID: "b243ef81-e040-408a-96e6-6ce5c7df25af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:25:52.335531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.335460 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-token" pod "b243ef81-e040-408a-96e6-6ce5c7df25af" (UID: "b243ef81-e040-408a-96e6-6ce5c7df25af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:25:52.434760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.434719 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:52.434760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.434752 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:52.434760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.434764 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b243ef81-e040-408a-96e6-6ce5c7df25af-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:52.435058 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.434777 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn9g7\" (UniqueName: \"kubernetes.io/projected/b243ef81-e040-408a-96e6-6ce5c7df25af-kube-api-access-kn9g7\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:52.526667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.525534 5015 generic.go:356] "Generic (PLEG): container finished" podID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerID="5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe" exitCode=0 Nov 03 09:25:52.526667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.525657 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" event={"ID":"b243ef81-e040-408a-96e6-6ce5c7df25af","Type":"ContainerDied","Data":"5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe"} Nov 03 09:25:52.526667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.525746 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" event={"ID":"b243ef81-e040-408a-96e6-6ce5c7df25af","Type":"ContainerDied","Data":"3226f3068f9e609b8e2c9d1fa244fffd326c70ee369dbe89815460ca69c96598"} Nov 03 09:25:52.526667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.525783 5015 scope.go:122] "RemoveContainer" containerID="5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe" Nov 03 09:25:52.526667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.525991 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s" Nov 03 09:25:52.548707 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.547710 5015 scope.go:122] "RemoveContainer" containerID="5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe" Nov 03 09:25:52.548707 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:25:52.548058 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe\": container with ID starting with 5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe not found: ID does not exist" containerID="5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe" Nov 03 09:25:52.548707 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.548095 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe"} err="failed to get container status \"5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe\": rpc error: code = NotFound desc = could not find container \"5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe\": container with ID starting with 5583acde0a0601e434f8a54a2009fed4a7aa1ef86792058862ce413ec4ce8bbe not found: ID does not exist" Nov 03 09:25:52.712007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.711949 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s"] Nov 03 09:25:52.716970 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:52.716938 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5gs6s"] Nov 03 09:25:53.206739 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:25:53.206704 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" path="/var/lib/kubelet/pods/b243ef81-e040-408a-96e6-6ce5c7df25af/volumes" Nov 03 09:26:16.903602 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:26:16.903555 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:26:43.496786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:43.496692 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs"] Nov 03 09:26:43.497255 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:43.496999 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" containerID="cri-o://ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62" gracePeriod=30 Nov 03 09:26:46.905046 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:26:46.904998 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:26:47.238245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:47.238209 5015 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:26:47.238415 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:47.238277 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:26:48.771875 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.771796 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:26:48.831325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.831290 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-socket\") pod \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " Nov 03 09:26:48.831553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.831399 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istiod-ca-cert\") pod \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " Nov 03 09:26:48.831553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.831442 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-kube-api-access-fvp7k\" (UniqueName: \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-kube-api-access-fvp7k\") pod \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " Nov 03 09:26:48.831700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.831561 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-data\") pod \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " Nov 03 09:26:48.831700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.831612 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-envoy\") pod \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " Nov 03 09:26:48.831700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.831645 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-socket" pod "dfbc2cb6-1360-456b-8c90-24f83fbdebd9" (UID: "dfbc2cb6-1360-456b-8c90-24f83fbdebd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:26:48.831700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.831689 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-credential-socket\") pod \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " Nov 03 09:26:48.831903 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.831769 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-token\" (UniqueName: \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-token\") pod \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " Nov 03 09:26:48.831903 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.831803 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-certs\") pod \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " Nov 03 09:26:48.831903 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.831852 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-podinfo\") pod \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\" (UID: \"dfbc2cb6-1360-456b-8c90-24f83fbdebd9\") " Nov 03 09:26:48.832058 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.831913 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-data" pod "dfbc2cb6-1360-456b-8c90-24f83fbdebd9" (UID: "dfbc2cb6-1360-456b-8c90-24f83fbdebd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:26:48.832058 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.832006 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-credential-socket" pod "dfbc2cb6-1360-456b-8c90-24f83fbdebd9" (UID: "dfbc2cb6-1360-456b-8c90-24f83fbdebd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:26:48.832143 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.832114 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-certs" pod "dfbc2cb6-1360-456b-8c90-24f83fbdebd9" (UID: "dfbc2cb6-1360-456b-8c90-24f83fbdebd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:26:48.832344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.832312 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istiod-ca-cert" pod "dfbc2cb6-1360-456b-8c90-24f83fbdebd9" (UID: "dfbc2cb6-1360-456b-8c90-24f83fbdebd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:26:48.832464 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.832436 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:48.832464 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.832459 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:48.832464 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.832469 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:48.832464 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.832520 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:48.832464 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.832528 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:48.834459 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.834427 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-envoy" pod "dfbc2cb6-1360-456b-8c90-24f83fbdebd9" (UID: "dfbc2cb6-1360-456b-8c90-24f83fbdebd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:26:48.834795 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.834770 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-podinfo" pod "dfbc2cb6-1360-456b-8c90-24f83fbdebd9" (UID: "dfbc2cb6-1360-456b-8c90-24f83fbdebd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:26:48.834908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.834876 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-kube-api-access-fvp7k" pod "dfbc2cb6-1360-456b-8c90-24f83fbdebd9" (UID: "dfbc2cb6-1360-456b-8c90-24f83fbdebd9"). InnerVolumeSpecName "kube-api-access-fvp7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:26:48.835618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.835596 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-token" pod "dfbc2cb6-1360-456b-8c90-24f83fbdebd9" (UID: "dfbc2cb6-1360-456b-8c90-24f83fbdebd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:26:48.933949 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.933917 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvp7k\" (UniqueName: \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-kube-api-access-fvp7k\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:48.933949 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.933946 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:48.933949 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.933957 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:48.934208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:48.933966 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbc2cb6-1360-456b-8c90-24f83fbdebd9-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:49.007774 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:49.007742 5015 generic.go:356] "Generic (PLEG): container finished" podID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerID="ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62" exitCode=0 Nov 03 09:26:49.007974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:49.007794 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" Nov 03 09:26:49.007974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:49.007817 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" event={"ID":"dfbc2cb6-1360-456b-8c90-24f83fbdebd9","Type":"ContainerDied","Data":"ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62"} Nov 03 09:26:49.007974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:49.007874 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs" event={"ID":"dfbc2cb6-1360-456b-8c90-24f83fbdebd9","Type":"ContainerDied","Data":"85e84fbb1fb29f090790e892477e5871244d975ff2e34b650dffc9a701f6adb8"} Nov 03 09:26:49.007974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:49.007897 5015 scope.go:122] "RemoveContainer" containerID="ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62" Nov 03 09:26:49.020016 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:49.019999 5015 scope.go:122] "RemoveContainer" containerID="ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62" Nov 03 09:26:49.020354 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:26:49.020317 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62\": container with ID starting with ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62 not found: ID does not exist" containerID="ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62" Nov 03 09:26:49.020449 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:49.020364 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62"} err="failed to get container status \"ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62\": rpc error: code = NotFound desc = could not find container \"ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62\": container with ID starting with ccd35bb145aefc3bc5d6af19d7cc3c3ce9e7650748eb54f1891c2744b3225d62 not found: ID does not exist" Nov 03 09:26:49.182835 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:49.182730 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs"] Nov 03 09:26:49.188114 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:49.188083 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-2pwcs"] Nov 03 09:26:49.205900 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:26:49.205865 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" path="/var/lib/kubelet/pods/dfbc2cb6-1360-456b-8c90-24f83fbdebd9/volumes" Nov 03 09:27:16.902260 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:27:16.902224 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:27:26.855520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:26.855468 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz"] Nov 03 09:27:31.856634 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:31.856590 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" podUID="777f762c-f419-479f-a1d1-8cb613249b1e" containerName="payload-processing" containerID="cri-o://00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb" gracePeriod=25 Nov 03 09:27:32.132993 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.132957 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:27:32.298296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.298239 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777f762c-f419-479f-a1d1-8cb613249b1e-kube-api-access-dlh25\" (UniqueName: \"kubernetes.io/projected/777f762c-f419-479f-a1d1-8cb613249b1e-kube-api-access-dlh25\") pod \"777f762c-f419-479f-a1d1-8cb613249b1e\" (UID: \"777f762c-f419-479f-a1d1-8cb613249b1e\") " Nov 03 09:27:32.298533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.298515 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777f762c-f419-479f-a1d1-8cb613249b1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777f762c-f419-479f-a1d1-8cb613249b1e-plugins-config-volume\") pod \"777f762c-f419-479f-a1d1-8cb613249b1e\" (UID: \"777f762c-f419-479f-a1d1-8cb613249b1e\") " Nov 03 09:27:32.298935 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.298910 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777f762c-f419-479f-a1d1-8cb613249b1e-plugins-config-volume" pod "777f762c-f419-479f-a1d1-8cb613249b1e" (UID: "777f762c-f419-479f-a1d1-8cb613249b1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:27:32.299044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.299029 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777f762c-f419-479f-a1d1-8cb613249b1e-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:32.300722 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.300699 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777f762c-f419-479f-a1d1-8cb613249b1e-kube-api-access-dlh25" pod "777f762c-f419-479f-a1d1-8cb613249b1e" (UID: "777f762c-f419-479f-a1d1-8cb613249b1e"). InnerVolumeSpecName "kube-api-access-dlh25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:27:32.389170 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.389072 5015 generic.go:356] "Generic (PLEG): container finished" podID="777f762c-f419-479f-a1d1-8cb613249b1e" containerID="00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb" exitCode=0 Nov 03 09:27:32.389170 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.389152 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" event={"ID":"777f762c-f419-479f-a1d1-8cb613249b1e","Type":"ContainerDied","Data":"00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb"} Nov 03 09:27:32.389170 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.389161 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" Nov 03 09:27:32.389469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.389248 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz" event={"ID":"777f762c-f419-479f-a1d1-8cb613249b1e","Type":"ContainerDied","Data":"7d5a4ec2938aea702a40cc74455de9c49ade4082211f0a95354074102d4e0b02"} Nov 03 09:27:32.389469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.389263 5015 scope.go:122] "RemoveContainer" containerID="00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb" Nov 03 09:27:32.400813 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.400786 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlh25\" (UniqueName: \"kubernetes.io/projected/777f762c-f419-479f-a1d1-8cb613249b1e-kube-api-access-dlh25\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:32.401670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.401651 5015 scope.go:122] "RemoveContainer" containerID="00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb" Nov 03 09:27:32.401987 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:27:32.401965 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb\": container with ID starting with 00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb not found: ID does not exist" containerID="00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb" Nov 03 09:27:32.402067 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.401998 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb"} err="failed to get container status \"00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb\": rpc error: code = NotFound desc = could not find container \"00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb\": container with ID starting with 00e106616d8183eb14735ae830d2a062f78a9dddf60fb9386aac00fd8acf62cb not found: ID does not exist" Nov 03 09:27:32.560037 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.560009 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz"] Nov 03 09:27:32.565836 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:32.565808 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2fc57e-67fcd65d8f-7vhhz"] Nov 03 09:27:33.205964 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:33.205936 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777f762c-f419-479f-a1d1-8cb613249b1e" path="/var/lib/kubelet/pods/777f762c-f419-479f-a1d1-8cb613249b1e/volumes" Nov 03 09:27:46.903228 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:27:46.903185 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:27:47.706586 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:47.706554 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx"] Nov 03 09:27:47.706775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:47.706760 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" podUID="9c41adb7-c695-4531-8811-86e0c73858da" containerName="istio-proxy" containerID="cri-o://d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571" gracePeriod=30 Nov 03 09:27:53.018276 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.018239 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:27:53.168354 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.168312 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-credential-socket\") pod \"9c41adb7-c695-4531-8811-86e0c73858da\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " Nov 03 09:27:53.168354 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.168358 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-kube-api-access-scjld\" (UniqueName: \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-kube-api-access-scjld\") pod \"9c41adb7-c695-4531-8811-86e0c73858da\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " Nov 03 09:27:53.168645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.168403 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c41adb7-c695-4531-8811-86e0c73858da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c41adb7-c695-4531-8811-86e0c73858da-istiod-ca-cert\") pod \"9c41adb7-c695-4531-8811-86e0c73858da\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " Nov 03 09:27:53.168645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.168427 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-istio-token\" (UniqueName: \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-istio-token\") pod \"9c41adb7-c695-4531-8811-86e0c73858da\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " Nov 03 09:27:53.168645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.168509 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-certs\") pod \"9c41adb7-c695-4531-8811-86e0c73858da\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " Nov 03 09:27:53.168645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.168568 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-envoy\") pod \"9c41adb7-c695-4531-8811-86e0c73858da\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " Nov 03 09:27:53.168880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.168647 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-socket\") pod \"9c41adb7-c695-4531-8811-86e0c73858da\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " Nov 03 09:27:53.168880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.168847 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c41adb7-c695-4531-8811-86e0c73858da-istiod-ca-cert" pod "9c41adb7-c695-4531-8811-86e0c73858da" (UID: "9c41adb7-c695-4531-8811-86e0c73858da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:27:53.169011 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.168945 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-credential-socket" pod "9c41adb7-c695-4531-8811-86e0c73858da" (UID: "9c41adb7-c695-4531-8811-86e0c73858da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:53.169011 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.168962 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-socket" pod "9c41adb7-c695-4531-8811-86e0c73858da" (UID: "9c41adb7-c695-4531-8811-86e0c73858da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:53.169011 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.168977 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-certs" pod "9c41adb7-c695-4531-8811-86e0c73858da" (UID: "9c41adb7-c695-4531-8811-86e0c73858da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:53.169299 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.169275 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c41adb7-c695-4531-8811-86e0c73858da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c41adb7-c695-4531-8811-86e0c73858da-istio-podinfo\") pod \"9c41adb7-c695-4531-8811-86e0c73858da\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " Nov 03 09:27:53.169421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.169372 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-data\") pod \"9c41adb7-c695-4531-8811-86e0c73858da\" (UID: \"9c41adb7-c695-4531-8811-86e0c73858da\") " Nov 03 09:27:53.169761 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.169733 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-data" pod "9c41adb7-c695-4531-8811-86e0c73858da" (UID: "9c41adb7-c695-4531-8811-86e0c73858da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:53.170246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.170218 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.170361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.170251 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.170361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.170266 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.170361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.170280 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.170361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.170295 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c41adb7-c695-4531-8811-86e0c73858da-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.171386 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.171363 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-kube-api-access-scjld" pod "9c41adb7-c695-4531-8811-86e0c73858da" (UID: "9c41adb7-c695-4531-8811-86e0c73858da"). InnerVolumeSpecName "kube-api-access-scjld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:27:53.171468 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.171406 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-envoy" pod "9c41adb7-c695-4531-8811-86e0c73858da" (UID: "9c41adb7-c695-4531-8811-86e0c73858da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:53.172143 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.172121 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-istio-token" pod "9c41adb7-c695-4531-8811-86e0c73858da" (UID: "9c41adb7-c695-4531-8811-86e0c73858da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:27:53.172312 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.172293 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c41adb7-c695-4531-8811-86e0c73858da-istio-podinfo" pod "9c41adb7-c695-4531-8811-86e0c73858da" (UID: "9c41adb7-c695-4531-8811-86e0c73858da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:27:53.271622 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.271527 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.271622 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.271556 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c41adb7-c695-4531-8811-86e0c73858da-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.271622 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.271568 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c41adb7-c695-4531-8811-86e0c73858da-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.271622 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.271577 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scjld\" (UniqueName: \"kubernetes.io/projected/9c41adb7-c695-4531-8811-86e0c73858da-kube-api-access-scjld\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.552283 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.552182 5015 generic.go:356] "Generic (PLEG): container finished" podID="9c41adb7-c695-4531-8811-86e0c73858da" containerID="d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571" exitCode=0 Nov 03 09:27:53.552283 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.552251 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" Nov 03 09:27:53.552536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.552271 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" event={"ID":"9c41adb7-c695-4531-8811-86e0c73858da","Type":"ContainerDied","Data":"d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571"} Nov 03 09:27:53.552536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.552335 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx" event={"ID":"9c41adb7-c695-4531-8811-86e0c73858da","Type":"ContainerDied","Data":"70955dbe94aafd6d859fefa85bac11d946a904d92871e786fed451ade78396cd"} Nov 03 09:27:53.552536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.552359 5015 scope.go:122] "RemoveContainer" containerID="d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571" Nov 03 09:27:53.564604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.564574 5015 scope.go:122] "RemoveContainer" containerID="d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571" Nov 03 09:27:53.564938 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:27:53.564917 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571\": container with ID starting with d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571 not found: ID does not exist" containerID="d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571" Nov 03 09:27:53.565013 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.564954 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571"} err="failed to get container status \"d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571\": rpc error: code = NotFound desc = could not find container \"d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571\": container with ID starting with d284b0ccadf9f757818808b185ab7897d7034cfa7e2d4388483e05cd99f91571 not found: ID does not exist" Nov 03 09:27:53.723192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.723154 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx"] Nov 03 09:27:53.729338 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:53.729308 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-4mklx"] Nov 03 09:27:55.208596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:55.208551 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c41adb7-c695-4531-8811-86e0c73858da" path="/var/lib/kubelet/pods/9c41adb7-c695-4531-8811-86e0c73858da/volumes" Nov 03 09:27:59.087710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:59.087672 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj"] Nov 03 09:27:59.088183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:27:59.087926 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" podUID="b7a4006d-88b5-4965-8738-1bd170cc4ede" containerName="istio-proxy" containerID="cri-o://dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302" gracePeriod=30 Nov 03 09:28:04.381611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.381535 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:28:04.420710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.420680 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-credential-socket\") pod \"b7a4006d-88b5-4965-8738-1bd170cc4ede\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " Nov 03 09:28:04.420842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.420741 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-data\") pod \"b7a4006d-88b5-4965-8738-1bd170cc4ede\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " Nov 03 09:28:04.420842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.420766 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-socket\") pod \"b7a4006d-88b5-4965-8738-1bd170cc4ede\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " Nov 03 09:28:04.420842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.420784 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-certs\") pod \"b7a4006d-88b5-4965-8738-1bd170cc4ede\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " Nov 03 09:28:04.420842 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.420808 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-kube-api-access-lkc2n\" (UniqueName: \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-kube-api-access-lkc2n\") pod \"b7a4006d-88b5-4965-8738-1bd170cc4ede\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " Nov 03 09:28:04.421069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.420910 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-token\" (UniqueName: \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-token\") pod \"b7a4006d-88b5-4965-8738-1bd170cc4ede\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " Nov 03 09:28:04.421069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.420951 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-envoy\") pod \"b7a4006d-88b5-4965-8738-1bd170cc4ede\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " Nov 03 09:28:04.421069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.420986 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-credential-socket" pod "b7a4006d-88b5-4965-8738-1bd170cc4ede" (UID: "b7a4006d-88b5-4965-8738-1bd170cc4ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:28:04.421069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.421030 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a4006d-88b5-4965-8738-1bd170cc4ede-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a4006d-88b5-4965-8738-1bd170cc4ede-istiod-ca-cert\") pod \"b7a4006d-88b5-4965-8738-1bd170cc4ede\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " Nov 03 09:28:04.421069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.421039 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-socket" pod "b7a4006d-88b5-4965-8738-1bd170cc4ede" (UID: "b7a4006d-88b5-4965-8738-1bd170cc4ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:28:04.421309 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.421089 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-podinfo\") pod \"b7a4006d-88b5-4965-8738-1bd170cc4ede\" (UID: \"b7a4006d-88b5-4965-8738-1bd170cc4ede\") " Nov 03 09:28:04.421309 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.421143 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-certs" pod "b7a4006d-88b5-4965-8738-1bd170cc4ede" (UID: "b7a4006d-88b5-4965-8738-1bd170cc4ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:28:04.421426 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.421336 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-data" pod "b7a4006d-88b5-4965-8738-1bd170cc4ede" (UID: "b7a4006d-88b5-4965-8738-1bd170cc4ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:28:04.421535 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.421514 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a4006d-88b5-4965-8738-1bd170cc4ede-istiod-ca-cert" pod "b7a4006d-88b5-4965-8738-1bd170cc4ede" (UID: "b7a4006d-88b5-4965-8738-1bd170cc4ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:28:04.421732 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.421709 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:04.421847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.421735 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:04.421847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.421748 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:04.421847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.421761 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:04.421847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.421773 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a4006d-88b5-4965-8738-1bd170cc4ede-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:04.423404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.423376 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-envoy" pod "b7a4006d-88b5-4965-8738-1bd170cc4ede" (UID: "b7a4006d-88b5-4965-8738-1bd170cc4ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:28:04.423404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.423387 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-token" pod "b7a4006d-88b5-4965-8738-1bd170cc4ede" (UID: "b7a4006d-88b5-4965-8738-1bd170cc4ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:28:04.423856 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.423832 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-kube-api-access-lkc2n" pod "b7a4006d-88b5-4965-8738-1bd170cc4ede" (UID: "b7a4006d-88b5-4965-8738-1bd170cc4ede"). InnerVolumeSpecName "kube-api-access-lkc2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:28:04.423932 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.423834 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-podinfo" pod "b7a4006d-88b5-4965-8738-1bd170cc4ede" (UID: "b7a4006d-88b5-4965-8738-1bd170cc4ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:28:04.523149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.523115 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkc2n\" (UniqueName: \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-kube-api-access-lkc2n\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:04.523149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.523145 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:04.523149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.523158 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:04.523397 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.523167 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a4006d-88b5-4965-8738-1bd170cc4ede-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:04.651568 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.651445 5015 generic.go:356] "Generic (PLEG): container finished" podID="b7a4006d-88b5-4965-8738-1bd170cc4ede" containerID="dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302" exitCode=0 Nov 03 09:28:04.651568 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.651523 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" Nov 03 09:28:04.651568 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.651539 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" event={"ID":"b7a4006d-88b5-4965-8738-1bd170cc4ede","Type":"ContainerDied","Data":"dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302"} Nov 03 09:28:04.651880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.651594 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj" event={"ID":"b7a4006d-88b5-4965-8738-1bd170cc4ede","Type":"ContainerDied","Data":"3b9e65182973790b112fd8493175d159e27ab4c589a5fe6e21a378faf4529af6"} Nov 03 09:28:04.651880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.651625 5015 scope.go:122] "RemoveContainer" containerID="dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302" Nov 03 09:28:04.665893 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.665874 5015 scope.go:122] "RemoveContainer" containerID="dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302" Nov 03 09:28:04.666351 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:28:04.666325 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302\": container with ID starting with dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302 not found: ID does not exist" containerID="dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302" Nov 03 09:28:04.666412 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.666365 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302"} err="failed to get container status \"dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302\": rpc error: code = NotFound desc = could not find container \"dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302\": container with ID starting with dd9ec005c6ea2928e505fb17c0b698e5943fd8c9929ee8cd852f14af11f12302 not found: ID does not exist" Nov 03 09:28:04.824411 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.824372 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj"] Nov 03 09:28:04.829627 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:04.829602 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cfgrj"] Nov 03 09:28:05.205908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:28:05.205833 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a4006d-88b5-4965-8738-1bd170cc4ede" path="/var/lib/kubelet/pods/b7a4006d-88b5-4965-8738-1bd170cc4ede/volumes" Nov 03 09:28:16.906828 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:28:16.906782 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:28:46.907934 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:28:46.907873 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:29:02.686697 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:02.686657 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7"] Nov 03 09:29:02.687211 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:02.686861 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" containerID="cri-o://8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465" gracePeriod=30 Nov 03 09:29:07.981329 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:07.981293 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:29:08.035922 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.035889 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-socket\") pod \"dcf17300-8bf7-445d-b29d-30774666a981\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " Nov 03 09:29:08.036079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.035937 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-kube-api-access-x8w6f\" (UniqueName: \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-kube-api-access-x8w6f\") pod \"dcf17300-8bf7-445d-b29d-30774666a981\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " Nov 03 09:29:08.036079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036039 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-istio-token\" (UniqueName: \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-istio-token\") pod \"dcf17300-8bf7-445d-b29d-30774666a981\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " Nov 03 09:29:08.036177 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036080 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-data\") pod \"dcf17300-8bf7-445d-b29d-30774666a981\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " Nov 03 09:29:08.036236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036176 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-envoy\") pod \"dcf17300-8bf7-445d-b29d-30774666a981\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " Nov 03 09:29:08.036236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036187 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-socket" pod "dcf17300-8bf7-445d-b29d-30774666a981" (UID: "dcf17300-8bf7-445d-b29d-30774666a981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:08.036236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036212 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcf17300-8bf7-445d-b29d-30774666a981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf17300-8bf7-445d-b29d-30774666a981-istio-podinfo\") pod \"dcf17300-8bf7-445d-b29d-30774666a981\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " Nov 03 09:29:08.036410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036254 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-certs\") pod \"dcf17300-8bf7-445d-b29d-30774666a981\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " Nov 03 09:29:08.036410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036292 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-credential-socket\") pod \"dcf17300-8bf7-445d-b29d-30774666a981\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " Nov 03 09:29:08.036410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036332 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf17300-8bf7-445d-b29d-30774666a981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf17300-8bf7-445d-b29d-30774666a981-istiod-ca-cert\") pod \"dcf17300-8bf7-445d-b29d-30774666a981\" (UID: \"dcf17300-8bf7-445d-b29d-30774666a981\") " Nov 03 09:29:08.036632 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036444 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-data" pod "dcf17300-8bf7-445d-b29d-30774666a981" (UID: "dcf17300-8bf7-445d-b29d-30774666a981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:08.036716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036691 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-certs" pod "dcf17300-8bf7-445d-b29d-30774666a981" (UID: "dcf17300-8bf7-445d-b29d-30774666a981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:08.036889 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036848 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-credential-socket" pod "dcf17300-8bf7-445d-b29d-30774666a981" (UID: "dcf17300-8bf7-445d-b29d-30774666a981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:08.036966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036951 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:08.037054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036973 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:08.037054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.036987 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:08.037054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.037013 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf17300-8bf7-445d-b29d-30774666a981-istiod-ca-cert" pod "dcf17300-8bf7-445d-b29d-30774666a981" (UID: "dcf17300-8bf7-445d-b29d-30774666a981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:29:08.038752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.038728 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-envoy" pod "dcf17300-8bf7-445d-b29d-30774666a981" (UID: "dcf17300-8bf7-445d-b29d-30774666a981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:08.038752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.038740 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcf17300-8bf7-445d-b29d-30774666a981-istio-podinfo" pod "dcf17300-8bf7-445d-b29d-30774666a981" (UID: "dcf17300-8bf7-445d-b29d-30774666a981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:29:08.038998 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.038981 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-istio-token" pod "dcf17300-8bf7-445d-b29d-30774666a981" (UID: "dcf17300-8bf7-445d-b29d-30774666a981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:29:08.039061 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.039021 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-kube-api-access-x8w6f" pod "dcf17300-8bf7-445d-b29d-30774666a981" (UID: "dcf17300-8bf7-445d-b29d-30774666a981"). InnerVolumeSpecName "kube-api-access-x8w6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:29:08.138697 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.138660 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:08.138697 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.138702 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf17300-8bf7-445d-b29d-30774666a981-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:08.138925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.138713 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf17300-8bf7-445d-b29d-30774666a981-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:08.138925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.138721 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf17300-8bf7-445d-b29d-30774666a981-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:08.138925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.138732 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8w6f\" (UniqueName: \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-kube-api-access-x8w6f\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:08.138925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.138743 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf17300-8bf7-445d-b29d-30774666a981-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:08.163133 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.163097 5015 generic.go:356] "Generic (PLEG): container finished" podID="dcf17300-8bf7-445d-b29d-30774666a981" containerID="8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465" exitCode=0 Nov 03 09:29:08.163284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.163173 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" event={"ID":"dcf17300-8bf7-445d-b29d-30774666a981","Type":"ContainerDied","Data":"8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465"} Nov 03 09:29:08.163284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.163197 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" Nov 03 09:29:08.163284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.163226 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7" event={"ID":"dcf17300-8bf7-445d-b29d-30774666a981","Type":"ContainerDied","Data":"e5f37ce62553d5b6cbd6e4d2dbff1276f7337c42e9b791626e58c9f6c8810d32"} Nov 03 09:29:08.163284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.163255 5015 scope.go:122] "RemoveContainer" containerID="8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465" Nov 03 09:29:08.177198 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.177170 5015 scope.go:122] "RemoveContainer" containerID="8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465" Nov 03 09:29:08.177561 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:29:08.177524 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465\": container with ID starting with 8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465 not found: ID does not exist" containerID="8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465" Nov 03 09:29:08.177660 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.177573 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465"} err="failed to get container status \"8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465\": rpc error: code = NotFound desc = could not find container \"8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465\": container with ID starting with 8acebe69262fd80478a57877e8bae36e7ac16c9f539545d1f0668389f7cec465 not found: ID does not exist" Nov 03 09:29:08.340563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.340529 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7"] Nov 03 09:29:08.346601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:08.346566 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-sq6c7"] Nov 03 09:29:09.205323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:09.205276 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf17300-8bf7-445d-b29d-30774666a981" path="/var/lib/kubelet/pods/dcf17300-8bf7-445d-b29d-30774666a981/volumes" Nov 03 09:29:14.698878 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:14.698842 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g"] Nov 03 09:29:14.699318 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:14.699101 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerName="istio-proxy" containerID="cri-o://ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112" gracePeriod=30 Nov 03 09:29:16.905255 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:29:16.905212 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:29:18.537222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:18.537179 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:29:18.537690 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:18.537242 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:29:20.017539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.017499 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:29:20.196262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.196227 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-kube-api-access-zs652\" (UniqueName: \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-kube-api-access-zs652\") pod \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " Nov 03 09:29:20.196462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.196357 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-podinfo\") pod \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " Nov 03 09:29:20.196462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.196397 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-envoy\") pod \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " Nov 03 09:29:20.196462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.196443 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-data\") pod \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " Nov 03 09:29:20.196725 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.196519 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-token\" (UniqueName: \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-token\") pod \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " Nov 03 09:29:20.196725 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.196577 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-credential-socket\") pod \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " Nov 03 09:29:20.196854 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.196785 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-data" pod "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" (UID: "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:20.196854 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.196822 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-credential-socket" pod "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" (UID: "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:20.198320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.197007 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-socket\") pod \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " Nov 03 09:29:20.198320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.197118 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-certs\") pod \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " Nov 03 09:29:20.198320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.197344 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istiod-ca-cert\") pod \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\" (UID: \"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e\") " Nov 03 09:29:20.198320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.197543 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-socket" pod "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" (UID: "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:20.198320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.197702 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-certs" pod "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" (UID: "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:20.198320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.197899 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istiod-ca-cert" pod "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" (UID: "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:29:20.198320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.198228 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:20.198320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.198250 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:20.198320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.198266 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:20.198320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.198279 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:20.198320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.198292 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:20.199318 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.199286 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-envoy" pod "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" (UID: "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:20.199408 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.199358 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-kube-api-access-zs652" pod "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" (UID: "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e"). InnerVolumeSpecName "kube-api-access-zs652". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:29:20.199511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.199456 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-token" pod "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" (UID: "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:29:20.199555 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.199504 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-podinfo" pod "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" (UID: "bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:29:20.276156 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.276073 5015 generic.go:356] "Generic (PLEG): container finished" podID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerID="ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112" exitCode=0 Nov 03 09:29:20.276156 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.276126 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" Nov 03 09:29:20.276334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.276164 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" event={"ID":"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e","Type":"ContainerDied","Data":"ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112"} Nov 03 09:29:20.276334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.276227 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g" event={"ID":"bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e","Type":"ContainerDied","Data":"837f147eb60c1c0402df0d41cbd8fa3b1c68f3c0f5e60a910fa039990f18edaf"} Nov 03 09:29:20.276334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.276256 5015 scope.go:122] "RemoveContainer" containerID="ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112" Nov 03 09:29:20.287902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.287883 5015 scope.go:122] "RemoveContainer" containerID="ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112" Nov 03 09:29:20.288189 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:29:20.288167 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112\": container with ID starting with ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112 not found: ID does not exist" containerID="ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112" Nov 03 09:29:20.288244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.288197 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112"} err="failed to get container status \"ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112\": rpc error: code = NotFound desc = could not find container \"ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112\": container with ID starting with ff1be99fbe5e2a62634ae67abe22c09ac684da0d5f47ab683948dfb88e0f5112 not found: ID does not exist" Nov 03 09:29:20.301344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.301315 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:20.301344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.301346 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:20.301604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.301361 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:20.301604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.301377 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs652\" (UniqueName: \"kubernetes.io/projected/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e-kube-api-access-zs652\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:20.448652 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.448594 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g"] Nov 03 09:29:20.475209 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:20.475168 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-jb57g"] Nov 03 09:29:21.205683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:21.205650 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" path="/var/lib/kubelet/pods/bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e/volumes" Nov 03 09:29:27.775447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:27.775414 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:29:27.792345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:27.792315 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:29:46.903100 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:29:46.903021 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:29:59.816963 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:29:59.816921 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv"] Nov 03 09:30:00.259434 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.259399 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9"] Nov 03 09:30:00.260159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260138 5015 cpu_manager.go:414] "containerMap: removing container" podUID="9c41adb7-c695-4531-8811-86e0c73858da" containerName="istio-proxy" Nov 03 09:30:00.260286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260163 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c41adb7-c695-4531-8811-86e0c73858da" containerName="istio-proxy" Nov 03 09:30:00.260286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260185 5015 cpu_manager.go:414] "containerMap: removing container" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" Nov 03 09:30:00.260286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260192 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" Nov 03 09:30:00.260286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260206 5015 cpu_manager.go:414] "containerMap: removing container" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerName="istio-proxy" Nov 03 09:30:00.260286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260214 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerName="istio-proxy" Nov 03 09:30:00.260286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260237 5015 cpu_manager.go:414] "containerMap: removing container" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerName="istio-proxy" Nov 03 09:30:00.260286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260244 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerName="istio-proxy" Nov 03 09:30:00.260286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260255 5015 cpu_manager.go:414] "containerMap: removing container" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerName="istio-proxy" Nov 03 09:30:00.260286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260262 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerName="istio-proxy" Nov 03 09:30:00.260286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260272 5015 cpu_manager.go:414] "containerMap: removing container" podUID="b7a4006d-88b5-4965-8738-1bd170cc4ede" containerName="istio-proxy" Nov 03 09:30:00.260286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260279 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a4006d-88b5-4965-8738-1bd170cc4ede" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260308 5015 cpu_manager.go:414] "containerMap: removing container" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260315 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260328 5015 cpu_manager.go:414] "containerMap: removing container" podUID="777f762c-f419-479f-a1d1-8cb613249b1e" containerName="payload-processing" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260336 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777f762c-f419-479f-a1d1-8cb613249b1e" containerName="payload-processing" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260385 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260395 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260579 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d233c76-e29e-43a0-abd0-212ce5df00a3" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260597 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c41adb7-c695-4531-8811-86e0c73858da" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260608 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbc2cb6-1360-456b-8c90-24f83fbdebd9" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260618 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9bbf13-18ef-4ba2-bff2-0ebb73acef2e" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260630 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e7beb7-7cc4-4dd7-8112-9c099f6afcaa" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260640 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="b243ef81-e040-408a-96e6-6ce5c7df25af" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260654 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a4006d-88b5-4965-8738-1bd170cc4ede" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260668 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf17300-8bf7-445d-b29d-30774666a981" containerName="istio-proxy" Nov 03 09:30:00.260908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.260675 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="777f762c-f419-479f-a1d1-8cb613249b1e" containerName="payload-processing" Nov 03 09:30:00.264767 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.264744 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5"] Nov 03 09:30:00.264913 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.264891 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" Nov 03 09:30:00.268980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.268958 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" Nov 03 09:30:00.274773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.274741 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65k2d\" (UniqueName: \"kubernetes.io/projected/9e02f4e5-e3f5-4f73-b9ca-62b503719697-kube-api-access-65k2d\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9\" (UID: \"9e02f4e5-e3f5-4f73-b9ca-62b503719697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" Nov 03 09:30:00.274941 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.274866 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e02f4e5-e3f5-4f73-b9ca-62b503719697-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9\" (UID: \"9e02f4e5-e3f5-4f73-b9ca-62b503719697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" Nov 03 09:30:00.290197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.290162 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm"] Nov 03 09:30:00.298251 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.298214 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" Nov 03 09:30:00.325594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.325561 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9"] Nov 03 09:30:00.329962 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.329933 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5"] Nov 03 09:30:00.373600 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.371647 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:30:00.378441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.376394 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm\" (UID: \"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" Nov 03 09:30:00.378441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.376459 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e02f4e5-e3f5-4f73-b9ca-62b503719697-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9\" (UID: \"9e02f4e5-e3f5-4f73-b9ca-62b503719697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" Nov 03 09:30:00.378441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.376570 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5\" (UID: \"ae69616b-adc9-4423-8d8d-5b6b6fb48a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" Nov 03 09:30:00.378441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.376671 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9gf\" (UniqueName: \"kubernetes.io/projected/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-kube-api-access-5h9gf\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5\" (UID: \"ae69616b-adc9-4423-8d8d-5b6b6fb48a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" Nov 03 09:30:00.378441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.376735 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65k2d\" (UniqueName: \"kubernetes.io/projected/9e02f4e5-e3f5-4f73-b9ca-62b503719697-kube-api-access-65k2d\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9\" (UID: \"9e02f4e5-e3f5-4f73-b9ca-62b503719697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" Nov 03 09:30:00.378441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.376767 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5zkl\" (UniqueName: \"kubernetes.io/projected/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-kube-api-access-d5zkl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm\" (UID: \"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" Nov 03 09:30:00.378441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.377509 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e02f4e5-e3f5-4f73-b9ca-62b503719697-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9\" (UID: \"9e02f4e5-e3f5-4f73-b9ca-62b503719697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" Nov 03 09:30:00.378441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.377971 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm"] Nov 03 09:30:00.440511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.440429 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65k2d\" (UniqueName: \"kubernetes.io/projected/9e02f4e5-e3f5-4f73-b9ca-62b503719697-kube-api-access-65k2d\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9\" (UID: \"9e02f4e5-e3f5-4f73-b9ca-62b503719697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" Nov 03 09:30:00.478546 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.478512 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9gf\" (UniqueName: \"kubernetes.io/projected/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-kube-api-access-5h9gf\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5\" (UID: \"ae69616b-adc9-4423-8d8d-5b6b6fb48a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" Nov 03 09:30:00.478735 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.478587 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5zkl\" (UniqueName: \"kubernetes.io/projected/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-kube-api-access-d5zkl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm\" (UID: \"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" Nov 03 09:30:00.478735 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.478630 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm\" (UID: \"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" Nov 03 09:30:00.478735 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.478726 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5\" (UID: \"ae69616b-adc9-4423-8d8d-5b6b6fb48a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" Nov 03 09:30:00.479311 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.479292 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm\" (UID: \"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" Nov 03 09:30:00.479358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.479338 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5\" (UID: \"ae69616b-adc9-4423-8d8d-5b6b6fb48a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" Nov 03 09:30:00.538129 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.538039 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5zkl\" (UniqueName: \"kubernetes.io/projected/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-kube-api-access-d5zkl\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm\" (UID: \"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" Nov 03 09:30:00.538306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.538252 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9gf\" (UniqueName: \"kubernetes.io/projected/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-kube-api-access-5h9gf\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5\" (UID: \"ae69616b-adc9-4423-8d8d-5b6b6fb48a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" Nov 03 09:30:00.594362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.594331 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" Nov 03 09:30:00.607690 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.607611 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" Nov 03 09:30:00.628693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:00.628660 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" Nov 03 09:30:01.038277 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:01.017684 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5"] Nov 03 09:30:01.060618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:01.060555 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9"] Nov 03 09:30:01.081725 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:30:01.080836 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e02f4e5_e3f5_4f73_b9ca_62b503719697.slice/crio-ed868eb7ccefc8056b4b5ceb6ad52ebbb7c14379179f66b8a0c6a2a1d2f756cd WatchSource:0}: Error finding container ed868eb7ccefc8056b4b5ceb6ad52ebbb7c14379179f66b8a0c6a2a1d2f756cd: Status 404 returned error can't find the container with id ed868eb7ccefc8056b4b5ceb6ad52ebbb7c14379179f66b8a0c6a2a1d2f756cd Nov 03 09:30:01.082420 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:01.082393 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm"] Nov 03 09:30:01.617041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:01.617004 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" event={"ID":"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b","Type":"ContainerStarted","Data":"9a3d1f5ff2f48834ccc6ea5f0f5d6e34d34b1397e0dee58795d6c3dd1c4eb0d3"} Nov 03 09:30:01.625354 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:01.625315 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" event={"ID":"9e02f4e5-e3f5-4f73-b9ca-62b503719697","Type":"ContainerStarted","Data":"ed868eb7ccefc8056b4b5ceb6ad52ebbb7c14379179f66b8a0c6a2a1d2f756cd"} Nov 03 09:30:01.627589 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:01.627465 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" event={"ID":"ae69616b-adc9-4423-8d8d-5b6b6fb48a81","Type":"ContainerStarted","Data":"16dce8deb13f4ba863e8142351c4623cdf2c10a8ab38908506dd791804b25458"} Nov 03 09:30:02.141513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:02.139186 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2"] Nov 03 09:30:02.636873 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:02.636830 5015 generic.go:356] "Generic (PLEG): container finished" podID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerID="2134135fd606aa9a969d9e3bd95ec1a2b604559e84d327ee0ff204d3f6cecbcc" exitCode=60 Nov 03 09:30:02.637077 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:02.636917 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" event={"ID":"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b","Type":"ContainerDied","Data":"2134135fd606aa9a969d9e3bd95ec1a2b604559e84d327ee0ff204d3f6cecbcc"} Nov 03 09:30:02.637227 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:02.637209 5015 scope.go:122] "RemoveContainer" containerID="2134135fd606aa9a969d9e3bd95ec1a2b604559e84d327ee0ff204d3f6cecbcc" Nov 03 09:30:02.639236 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:02.639212 5015 generic.go:356] "Generic (PLEG): container finished" podID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerID="7432fee260b9a2970b4003793dde0b09fc251cecd2b9699f94c22a7b0f5b7406" exitCode=60 Nov 03 09:30:02.639327 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:02.639301 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" event={"ID":"9e02f4e5-e3f5-4f73-b9ca-62b503719697","Type":"ContainerDied","Data":"7432fee260b9a2970b4003793dde0b09fc251cecd2b9699f94c22a7b0f5b7406"} Nov 03 09:30:02.639578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:02.639553 5015 scope.go:122] "RemoveContainer" containerID="7432fee260b9a2970b4003793dde0b09fc251cecd2b9699f94c22a7b0f5b7406" Nov 03 09:30:02.642418 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:02.642392 5015 generic.go:356] "Generic (PLEG): container finished" podID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerID="4d2811637364e26a611b93491ebbdd4578bf33de1b4c7689e27d6cf4a61c34a7" exitCode=60 Nov 03 09:30:02.642556 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:02.642462 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" event={"ID":"ae69616b-adc9-4423-8d8d-5b6b6fb48a81","Type":"ContainerDied","Data":"4d2811637364e26a611b93491ebbdd4578bf33de1b4c7689e27d6cf4a61c34a7"} Nov 03 09:30:02.642830 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:02.642716 5015 scope.go:122] "RemoveContainer" containerID="4d2811637364e26a611b93491ebbdd4578bf33de1b4c7689e27d6cf4a61c34a7" Nov 03 09:30:03.659529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:03.656466 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" event={"ID":"9e02f4e5-e3f5-4f73-b9ca-62b503719697","Type":"ContainerStarted","Data":"61f4ea4d41d9fe249249ee794aedd9120740acaac6a721edf3f7418fc9145bba"} Nov 03 09:30:03.660865 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:03.660833 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" event={"ID":"ae69616b-adc9-4423-8d8d-5b6b6fb48a81","Type":"ContainerStarted","Data":"0d52354f09ee5973480c6d02c7db1286eb2a0f510867bed80a294274eb6560d1"} Nov 03 09:30:03.661232 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:03.661211 5015 scope.go:122] "RemoveContainer" containerID="0d52354f09ee5973480c6d02c7db1286eb2a0f510867bed80a294274eb6560d1" Nov 03 09:30:03.661587 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:03.661554 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5_odh-ai-gateway-infra(ae69616b-adc9-4423-8d8d-5b6b6fb48a81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" Nov 03 09:30:03.663176 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:03.663152 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" event={"ID":"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b","Type":"ContainerStarted","Data":"2b5bdeb6cef9896f1e302c3f182f4c0789720fa4447abdcf526c4e44663da9d7"} Nov 03 09:30:03.774512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:03.774443 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" podStartSLOduration=3.7744294419999997 podStartE2EDuration="3.774429442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:03.772159437 +0000 UTC m=+3036.697101700" watchObservedRunningTime="2026-11-03 09:30:03.774429442 +0000 UTC m=+3036.699371672" Nov 03 09:30:04.004154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.004090 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" podStartSLOduration=4.004073444 podStartE2EDuration="4.004073444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:04.002766978 +0000 UTC m=+3036.927709196" watchObservedRunningTime="2026-11-03 09:30:04.004073444 +0000 UTC m=+3036.929015661" Nov 03 09:30:04.678269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.678232 5015 generic.go:356] "Generic (PLEG): container finished" podID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerID="2b5bdeb6cef9896f1e302c3f182f4c0789720fa4447abdcf526c4e44663da9d7" exitCode=60 Nov 03 09:30:04.678815 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.678372 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" event={"ID":"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b","Type":"ContainerDied","Data":"2b5bdeb6cef9896f1e302c3f182f4c0789720fa4447abdcf526c4e44663da9d7"} Nov 03 09:30:04.678815 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.678450 5015 scope.go:122] "RemoveContainer" containerID="2134135fd606aa9a969d9e3bd95ec1a2b604559e84d327ee0ff204d3f6cecbcc" Nov 03 09:30:04.678815 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.678642 5015 scope.go:122] "RemoveContainer" containerID="2b5bdeb6cef9896f1e302c3f182f4c0789720fa4447abdcf526c4e44663da9d7" Nov 03 09:30:04.679003 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:04.678964 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm_odh-ai-gateway-infra(5d4aa2d0-e62c-4f37-937e-ab210f4fe52b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" Nov 03 09:30:04.681814 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.681787 5015 generic.go:356] "Generic (PLEG): container finished" podID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerID="61f4ea4d41d9fe249249ee794aedd9120740acaac6a721edf3f7418fc9145bba" exitCode=60 Nov 03 09:30:04.681961 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.681830 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" event={"ID":"9e02f4e5-e3f5-4f73-b9ca-62b503719697","Type":"ContainerDied","Data":"61f4ea4d41d9fe249249ee794aedd9120740acaac6a721edf3f7418fc9145bba"} Nov 03 09:30:04.682163 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.682147 5015 scope.go:122] "RemoveContainer" containerID="61f4ea4d41d9fe249249ee794aedd9120740acaac6a721edf3f7418fc9145bba" Nov 03 09:30:04.682440 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:04.682421 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9_odh-ai-gateway-infra(9e02f4e5-e3f5-4f73-b9ca-62b503719697)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" Nov 03 09:30:04.684050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.684027 5015 generic.go:356] "Generic (PLEG): container finished" podID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerID="0d52354f09ee5973480c6d02c7db1286eb2a0f510867bed80a294274eb6560d1" exitCode=60 Nov 03 09:30:04.684168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.684062 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" event={"ID":"ae69616b-adc9-4423-8d8d-5b6b6fb48a81","Type":"ContainerDied","Data":"0d52354f09ee5973480c6d02c7db1286eb2a0f510867bed80a294274eb6560d1"} Nov 03 09:30:04.684373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.684352 5015 scope.go:122] "RemoveContainer" containerID="0d52354f09ee5973480c6d02c7db1286eb2a0f510867bed80a294274eb6560d1" Nov 03 09:30:04.684657 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:04.684638 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5_odh-ai-gateway-infra(ae69616b-adc9-4423-8d8d-5b6b6fb48a81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" Nov 03 09:30:04.696358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.696337 5015 scope.go:122] "RemoveContainer" containerID="7432fee260b9a2970b4003793dde0b09fc251cecd2b9699f94c22a7b0f5b7406" Nov 03 09:30:04.714569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:04.714546 5015 scope.go:122] "RemoveContainer" containerID="4d2811637364e26a611b93491ebbdd4578bf33de1b4c7689e27d6cf4a61c34a7" Nov 03 09:30:05.695626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:05.695594 5015 scope.go:122] "RemoveContainer" containerID="2b5bdeb6cef9896f1e302c3f182f4c0789720fa4447abdcf526c4e44663da9d7" Nov 03 09:30:05.696094 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:05.695951 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm_odh-ai-gateway-infra(5d4aa2d0-e62c-4f37-937e-ab210f4fe52b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" Nov 03 09:30:05.698325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:05.698298 5015 scope.go:122] "RemoveContainer" containerID="61f4ea4d41d9fe249249ee794aedd9120740acaac6a721edf3f7418fc9145bba" Nov 03 09:30:05.698635 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:05.698610 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9_odh-ai-gateway-infra(9e02f4e5-e3f5-4f73-b9ca-62b503719697)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" Nov 03 09:30:05.700121 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:05.700102 5015 scope.go:122] "RemoveContainer" containerID="0d52354f09ee5973480c6d02c7db1286eb2a0f510867bed80a294274eb6560d1" Nov 03 09:30:05.700353 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:05.700334 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5_odh-ai-gateway-infra(ae69616b-adc9-4423-8d8d-5b6b6fb48a81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" Nov 03 09:30:14.196767 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:14.196735 5015 scope.go:122] "RemoveContainer" containerID="61f4ea4d41d9fe249249ee794aedd9120740acaac6a721edf3f7418fc9145bba" Nov 03 09:30:14.770840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:14.770804 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" event={"ID":"9e02f4e5-e3f5-4f73-b9ca-62b503719697","Type":"ContainerStarted","Data":"31f914dad6e7090b2658449fcde56e9f168720fc75df6ad46fff0845601f7c4a"} Nov 03 09:30:14.839382 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:14.839343 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" podUID="d1fb16c2-c2aa-474a-9567-b85f640c4110" containerName="main" containerID="cri-o://06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b" gracePeriod=45 Nov 03 09:30:15.125418 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.125391 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:30:15.211399 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.211369 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1fb16c2-c2aa-474a-9567-b85f640c4110-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb16c2-c2aa-474a-9567-b85f640c4110-tls-certs\") pod \"d1fb16c2-c2aa-474a-9567-b85f640c4110\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " Nov 03 09:30:15.211802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.211469 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1fb16c2-c2aa-474a-9567-b85f640c4110-kube-api-access-gddxf\" (UniqueName: \"kubernetes.io/projected/d1fb16c2-c2aa-474a-9567-b85f640c4110-kube-api-access-gddxf\") pod \"d1fb16c2-c2aa-474a-9567-b85f640c4110\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " Nov 03 09:30:15.211802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.211554 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-dshm\") pod \"d1fb16c2-c2aa-474a-9567-b85f640c4110\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " Nov 03 09:30:15.211802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.211604 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-tmp-dir\") pod \"d1fb16c2-c2aa-474a-9567-b85f640c4110\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " Nov 03 09:30:15.211802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.211639 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-model-cache\") pod \"d1fb16c2-c2aa-474a-9567-b85f640c4110\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " Nov 03 09:30:15.211802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.211706 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-home\") pod \"d1fb16c2-c2aa-474a-9567-b85f640c4110\" (UID: \"d1fb16c2-c2aa-474a-9567-b85f640c4110\") " Nov 03 09:30:15.212091 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.211892 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-tmp-dir" pod "d1fb16c2-c2aa-474a-9567-b85f640c4110" (UID: "d1fb16c2-c2aa-474a-9567-b85f640c4110"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:30:15.212091 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.212002 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-model-cache" pod "d1fb16c2-c2aa-474a-9567-b85f640c4110" (UID: "d1fb16c2-c2aa-474a-9567-b85f640c4110"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:30:15.212091 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.212073 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-home" pod "d1fb16c2-c2aa-474a-9567-b85f640c4110" (UID: "d1fb16c2-c2aa-474a-9567-b85f640c4110"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:30:15.212368 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.212348 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:15.212411 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.212375 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:15.212411 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.212391 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:15.213848 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.213822 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fb16c2-c2aa-474a-9567-b85f640c4110-tls-certs" pod "d1fb16c2-c2aa-474a-9567-b85f640c4110" (UID: "d1fb16c2-c2aa-474a-9567-b85f640c4110"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:30:15.214347 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.214321 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-dshm" pod "d1fb16c2-c2aa-474a-9567-b85f640c4110" (UID: "d1fb16c2-c2aa-474a-9567-b85f640c4110"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:30:15.214445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.214350 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fb16c2-c2aa-474a-9567-b85f640c4110-kube-api-access-gddxf" pod "d1fb16c2-c2aa-474a-9567-b85f640c4110" (UID: "d1fb16c2-c2aa-474a-9567-b85f640c4110"). InnerVolumeSpecName "kube-api-access-gddxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:15.314084 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.314053 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb16c2-c2aa-474a-9567-b85f640c4110-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:15.314084 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.314082 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gddxf\" (UniqueName: \"kubernetes.io/projected/d1fb16c2-c2aa-474a-9567-b85f640c4110-kube-api-access-gddxf\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:15.314084 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.314092 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb16c2-c2aa-474a-9567-b85f640c4110-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:15.785023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.783428 5015 generic.go:356] "Generic (PLEG): container finished" podID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerID="31f914dad6e7090b2658449fcde56e9f168720fc75df6ad46fff0845601f7c4a" exitCode=60 Nov 03 09:30:15.785023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.783573 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" event={"ID":"9e02f4e5-e3f5-4f73-b9ca-62b503719697","Type":"ContainerDied","Data":"31f914dad6e7090b2658449fcde56e9f168720fc75df6ad46fff0845601f7c4a"} Nov 03 09:30:15.785023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.783636 5015 scope.go:122] "RemoveContainer" containerID="61f4ea4d41d9fe249249ee794aedd9120740acaac6a721edf3f7418fc9145bba" Nov 03 09:30:15.785023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.784129 5015 scope.go:122] "RemoveContainer" containerID="31f914dad6e7090b2658449fcde56e9f168720fc75df6ad46fff0845601f7c4a" Nov 03 09:30:15.785023 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:15.784551 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9_odh-ai-gateway-infra(9e02f4e5-e3f5-4f73-b9ca-62b503719697)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" Nov 03 09:30:15.787805 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.787779 5015 generic.go:356] "Generic (PLEG): container finished" podID="d1fb16c2-c2aa-474a-9567-b85f640c4110" containerID="06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b" exitCode=0 Nov 03 09:30:15.787925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.787912 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" event={"ID":"d1fb16c2-c2aa-474a-9567-b85f640c4110","Type":"ContainerDied","Data":"06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b"} Nov 03 09:30:15.788006 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.787935 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" event={"ID":"d1fb16c2-c2aa-474a-9567-b85f640c4110","Type":"ContainerDied","Data":"429da9d8a3b36c9dabf3b0e58f61b2afa49f30477be88c3ddf7dcf4cf1c33aab"} Nov 03 09:30:15.788006 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.787998 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv" Nov 03 09:30:15.802872 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.802836 5015 scope.go:122] "RemoveContainer" containerID="06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b" Nov 03 09:30:15.812973 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.812956 5015 scope.go:122] "RemoveContainer" containerID="06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b" Nov 03 09:30:15.813278 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:15.813259 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b\": container with ID starting with 06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b not found: ID does not exist" containerID="06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b" Nov 03 09:30:15.813342 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.813292 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b"} err="failed to get container status \"06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b\": rpc error: code = NotFound desc = could not find container \"06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b\": container with ID starting with 06a04a248243556f20d2ef45c0a4c1a1fe2c75389a6f7bbff83d5ed0f810f81b not found: ID does not exist" Nov 03 09:30:15.900426 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:15.900390 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9"] Nov 03 09:30:16.027376 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.027344 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv"] Nov 03 09:30:16.033799 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.033774 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/auth-test-model-627e5f-kserve-854b76dc89-dmsxv"] Nov 03 09:30:16.068988 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.068914 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b"] Nov 03 09:30:16.069903 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.069883 5015 cpu_manager.go:414] "containerMap: removing container" podUID="d1fb16c2-c2aa-474a-9567-b85f640c4110" containerName="main" Nov 03 09:30:16.069951 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.069907 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1fb16c2-c2aa-474a-9567-b85f640c4110" containerName="main" Nov 03 09:30:16.070167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.070154 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1fb16c2-c2aa-474a-9567-b85f640c4110" containerName="main" Nov 03 09:30:16.074241 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.074219 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:30:16.124341 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.124310 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6b8abd-22ac-4661-a14e-a7262397e9cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b\" (UID: \"bd6b8abd-22ac-4661-a14e-a7262397e9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:30:16.124656 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.124377 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpwsk\" (UniqueName: \"kubernetes.io/projected/bd6b8abd-22ac-4661-a14e-a7262397e9cb-kube-api-access-mpwsk\") pod \"payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b\" (UID: \"bd6b8abd-22ac-4661-a14e-a7262397e9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:30:16.128896 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.128869 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-03a6e5\"" Nov 03 09:30:16.129054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.128998 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-03a6e5-dockercfg-2j8mq\"" Nov 03 09:30:16.129711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.129602 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b"] Nov 03 09:30:16.191519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.191434 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw"] Nov 03 09:30:16.197535 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.197507 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:30:16.225311 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.225278 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzqmd\" (UniqueName: \"kubernetes.io/projected/6d7a2217-86bb-4483-acb5-9125efc1f71d-kube-api-access-mzqmd\") pod \"payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw\" (UID: \"6d7a2217-86bb-4483-acb5-9125efc1f71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:30:16.225740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.225384 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7a2217-86bb-4483-acb5-9125efc1f71d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw\" (UID: \"6d7a2217-86bb-4483-acb5-9125efc1f71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:30:16.225740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.225715 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6b8abd-22ac-4661-a14e-a7262397e9cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b\" (UID: \"bd6b8abd-22ac-4661-a14e-a7262397e9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:30:16.225894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.225872 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwsk\" (UniqueName: \"kubernetes.io/projected/bd6b8abd-22ac-4661-a14e-a7262397e9cb-kube-api-access-mpwsk\") pod \"payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b\" (UID: \"bd6b8abd-22ac-4661-a14e-a7262397e9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:30:16.226633 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.226609 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6b8abd-22ac-4661-a14e-a7262397e9cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b\" (UID: \"bd6b8abd-22ac-4661-a14e-a7262397e9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:30:16.230197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.230169 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw"] Nov 03 09:30:16.285725 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.285685 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwsk\" (UniqueName: \"kubernetes.io/projected/bd6b8abd-22ac-4661-a14e-a7262397e9cb-kube-api-access-mpwsk\") pod \"payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b\" (UID: \"bd6b8abd-22ac-4661-a14e-a7262397e9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:30:16.327168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.327089 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzqmd\" (UniqueName: \"kubernetes.io/projected/6d7a2217-86bb-4483-acb5-9125efc1f71d-kube-api-access-mzqmd\") pod \"payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw\" (UID: \"6d7a2217-86bb-4483-acb5-9125efc1f71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:30:16.327168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.327132 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7a2217-86bb-4483-acb5-9125efc1f71d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw\" (UID: \"6d7a2217-86bb-4483-acb5-9125efc1f71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:30:16.327891 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.327872 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7a2217-86bb-4483-acb5-9125efc1f71d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw\" (UID: \"6d7a2217-86bb-4483-acb5-9125efc1f71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:30:16.387050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.387016 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzqmd\" (UniqueName: \"kubernetes.io/projected/6d7a2217-86bb-4483-acb5-9125efc1f71d-kube-api-access-mzqmd\") pod \"payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw\" (UID: \"6d7a2217-86bb-4483-acb5-9125efc1f71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:30:16.391863 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.391840 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:30:16.512711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.512678 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:30:16.786802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.786701 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b"] Nov 03 09:30:16.787729 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:30:16.787704 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6b8abd_22ac_4661_a14e_a7262397e9cb.slice/crio-acb5e646ccec59351ac0643effc3a5f1178e007e5e79c80379616dd736941d69 WatchSource:0}: Error finding container acb5e646ccec59351ac0643effc3a5f1178e007e5e79c80379616dd736941d69: Status 404 returned error can't find the container with id acb5e646ccec59351ac0643effc3a5f1178e007e5e79c80379616dd736941d69 Nov 03 09:30:16.803832 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.803799 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" event={"ID":"bd6b8abd-22ac-4661-a14e-a7262397e9cb","Type":"ContainerStarted","Data":"acb5e646ccec59351ac0643effc3a5f1178e007e5e79c80379616dd736941d69"} Nov 03 09:30:16.920226 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.920142 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw"] Nov 03 09:30:16.927845 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:16.924983 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:30:16.963279 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:16.963252 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" Nov 03 09:30:17.035179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.035151 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e02f4e5-e3f5-4f73-b9ca-62b503719697-kube-api-access-65k2d\" (UniqueName: \"kubernetes.io/projected/9e02f4e5-e3f5-4f73-b9ca-62b503719697-kube-api-access-65k2d\") pod \"9e02f4e5-e3f5-4f73-b9ca-62b503719697\" (UID: \"9e02f4e5-e3f5-4f73-b9ca-62b503719697\") " Nov 03 09:30:17.035308 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.035204 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e02f4e5-e3f5-4f73-b9ca-62b503719697-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e02f4e5-e3f5-4f73-b9ca-62b503719697-maas-api-service-ca\") pod \"9e02f4e5-e3f5-4f73-b9ca-62b503719697\" (UID: \"9e02f4e5-e3f5-4f73-b9ca-62b503719697\") " Nov 03 09:30:17.035670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.035636 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e02f4e5-e3f5-4f73-b9ca-62b503719697-maas-api-service-ca" pod "9e02f4e5-e3f5-4f73-b9ca-62b503719697" (UID: "9e02f4e5-e3f5-4f73-b9ca-62b503719697"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:30:17.037690 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.037662 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e02f4e5-e3f5-4f73-b9ca-62b503719697-kube-api-access-65k2d" pod "9e02f4e5-e3f5-4f73-b9ca-62b503719697" (UID: "9e02f4e5-e3f5-4f73-b9ca-62b503719697"). InnerVolumeSpecName "kube-api-access-65k2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:17.136466 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.136392 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65k2d\" (UniqueName: \"kubernetes.io/projected/9e02f4e5-e3f5-4f73-b9ca-62b503719697-kube-api-access-65k2d\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:17.136466 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.136421 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e02f4e5-e3f5-4f73-b9ca-62b503719697-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:17.173605 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.173567 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" podUID="89525c49-48f6-4822-9a33-159ede94f8af" containerName="main" containerID="cri-o://2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354" gracePeriod=45 Nov 03 09:30:17.201407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.201383 5015 scope.go:122] "RemoveContainer" containerID="2b5bdeb6cef9896f1e302c3f182f4c0789720fa4447abdcf526c4e44663da9d7" Nov 03 09:30:17.206701 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.206675 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1fb16c2-c2aa-474a-9567-b85f640c4110" path="/var/lib/kubelet/pods/d1fb16c2-c2aa-474a-9567-b85f640c4110/volumes" Nov 03 09:30:17.477645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.477620 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:30:17.540292 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.540252 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-tmp-dir\") pod \"89525c49-48f6-4822-9a33-159ede94f8af\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " Nov 03 09:30:17.540495 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.540351 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-model-cache\") pod \"89525c49-48f6-4822-9a33-159ede94f8af\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " Nov 03 09:30:17.540495 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.540389 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89525c49-48f6-4822-9a33-159ede94f8af-kube-api-access-8dmzs\" (UniqueName: \"kubernetes.io/projected/89525c49-48f6-4822-9a33-159ede94f8af-kube-api-access-8dmzs\") pod \"89525c49-48f6-4822-9a33-159ede94f8af\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " Nov 03 09:30:17.540495 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.540426 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-dshm\") pod \"89525c49-48f6-4822-9a33-159ede94f8af\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " Nov 03 09:30:17.540684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.540516 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-home\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-home\") pod \"89525c49-48f6-4822-9a33-159ede94f8af\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " Nov 03 09:30:17.540684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.540558 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-tmp-dir" pod "89525c49-48f6-4822-9a33-159ede94f8af" (UID: "89525c49-48f6-4822-9a33-159ede94f8af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:30:17.540684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.540617 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89525c49-48f6-4822-9a33-159ede94f8af-tls-certs\" (UniqueName: \"kubernetes.io/secret/89525c49-48f6-4822-9a33-159ede94f8af-tls-certs\") pod \"89525c49-48f6-4822-9a33-159ede94f8af\" (UID: \"89525c49-48f6-4822-9a33-159ede94f8af\") " Nov 03 09:30:17.540861 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.540699 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-model-cache" pod "89525c49-48f6-4822-9a33-159ede94f8af" (UID: "89525c49-48f6-4822-9a33-159ede94f8af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:30:17.540959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.540923 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-home" pod "89525c49-48f6-4822-9a33-159ede94f8af" (UID: "89525c49-48f6-4822-9a33-159ede94f8af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:30:17.541141 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.541120 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:17.541226 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.541144 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:17.541226 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.541158 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:17.543344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.543318 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89525c49-48f6-4822-9a33-159ede94f8af-tls-certs" pod "89525c49-48f6-4822-9a33-159ede94f8af" (UID: "89525c49-48f6-4822-9a33-159ede94f8af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:30:17.543617 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.543582 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-dshm" pod "89525c49-48f6-4822-9a33-159ede94f8af" (UID: "89525c49-48f6-4822-9a33-159ede94f8af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:30:17.543698 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.543628 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89525c49-48f6-4822-9a33-159ede94f8af-kube-api-access-8dmzs" pod "89525c49-48f6-4822-9a33-159ede94f8af" (UID: "89525c49-48f6-4822-9a33-159ede94f8af"). InnerVolumeSpecName "kube-api-access-8dmzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:17.642407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.642372 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89525c49-48f6-4822-9a33-159ede94f8af-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:17.642596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.642413 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dmzs\" (UniqueName: \"kubernetes.io/projected/89525c49-48f6-4822-9a33-159ede94f8af-kube-api-access-8dmzs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:17.642596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.642428 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89525c49-48f6-4822-9a33-159ede94f8af-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:17.812002 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.811971 5015 generic.go:356] "Generic (PLEG): container finished" podID="89525c49-48f6-4822-9a33-159ede94f8af" containerID="2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354" exitCode=0 Nov 03 09:30:17.812193 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.812032 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" Nov 03 09:30:17.812193 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.812051 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" event={"ID":"89525c49-48f6-4822-9a33-159ede94f8af","Type":"ContainerDied","Data":"2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354"} Nov 03 09:30:17.812193 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.812096 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2" event={"ID":"89525c49-48f6-4822-9a33-159ede94f8af","Type":"ContainerDied","Data":"edb6dab8c6a75523258703836d3c2443fff27b85902ae420e91551d092acf05f"} Nov 03 09:30:17.812193 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.812117 5015 scope.go:122] "RemoveContainer" containerID="2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354" Nov 03 09:30:17.814116 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.814092 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" event={"ID":"6d7a2217-86bb-4483-acb5-9125efc1f71d","Type":"ContainerStarted","Data":"d6d5af0b79e5d3a9ed5b14622d3f118ebdfbfa2ced39fd994554d03bc14546d1"} Nov 03 09:30:17.814252 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.814183 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:30:17.814252 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.814203 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" event={"ID":"6d7a2217-86bb-4483-acb5-9125efc1f71d","Type":"ContainerStarted","Data":"149a2db12ebd5bfbb2f4aeb0a01a92ab0643265d107a1987c0d960cd3c53d6c2"} Nov 03 09:30:17.816271 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.816250 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" event={"ID":"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b","Type":"ContainerStarted","Data":"6c17f7f89c9f58219faa8753ef456cfd3a77b6a65742f268c611b849391230d7"} Nov 03 09:30:17.817906 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.817888 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" Nov 03 09:30:17.818028 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.817912 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9" event={"ID":"9e02f4e5-e3f5-4f73-b9ca-62b503719697","Type":"ContainerDied","Data":"ed868eb7ccefc8056b4b5ceb6ad52ebbb7c14379179f66b8a0c6a2a1d2f756cd"} Nov 03 09:30:17.820344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.820320 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" event={"ID":"bd6b8abd-22ac-4661-a14e-a7262397e9cb","Type":"ContainerStarted","Data":"bc1f60ef861a2e1f88ca55d9968c467d379899ab53bd1a0e385fbdf16c6bf7f7"} Nov 03 09:30:17.820469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.820455 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:30:17.825973 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.825957 5015 scope.go:122] "RemoveContainer" containerID="2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354" Nov 03 09:30:17.826244 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:17.826224 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354\": container with ID starting with 2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354 not found: ID does not exist" containerID="2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354" Nov 03 09:30:17.826281 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.826253 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354"} err="failed to get container status \"2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354\": rpc error: code = NotFound desc = could not find container \"2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354\": container with ID starting with 2eec76e245aa4d94cd87c8b01b1ebbc29c83f21fb691ab99f47dbc608a495354 not found: ID does not exist" Nov 03 09:30:17.826281 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:17.826270 5015 scope.go:122] "RemoveContainer" containerID="31f914dad6e7090b2658449fcde56e9f168720fc75df6ad46fff0845601f7c4a" Nov 03 09:30:18.097147 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.097056 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2"] Nov 03 09:30:18.100406 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.100383 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/auth-test-model-506e6d-kserve-cf9c44645-rhjf2"] Nov 03 09:30:18.197054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.197020 5015 scope.go:122] "RemoveContainer" containerID="0d52354f09ee5973480c6d02c7db1286eb2a0f510867bed80a294274eb6560d1" Nov 03 09:30:18.214630 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.214608 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9"] Nov 03 09:30:18.220050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.220026 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894970-cctq9"] Nov 03 09:30:18.336642 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.336586 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" podStartSLOduration=2.336571912 podStartE2EDuration="2.336571912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:18.332643454 +0000 UTC m=+3051.257585674" watchObservedRunningTime="2026-11-03 09:30:18.336571912 +0000 UTC m=+3051.261514129" Nov 03 09:30:18.463008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.462952 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" podStartSLOduration=2.4629372050000002 podStartE2EDuration="2.462937205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:18.460372655 +0000 UTC m=+3051.385314887" watchObservedRunningTime="2026-11-03 09:30:18.462937205 +0000 UTC m=+3051.387879421" Nov 03 09:30:18.836152 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.836119 5015 generic.go:356] "Generic (PLEG): container finished" podID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerID="6c17f7f89c9f58219faa8753ef456cfd3a77b6a65742f268c611b849391230d7" exitCode=60 Nov 03 09:30:18.836577 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.836214 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" event={"ID":"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b","Type":"ContainerDied","Data":"6c17f7f89c9f58219faa8753ef456cfd3a77b6a65742f268c611b849391230d7"} Nov 03 09:30:18.836577 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.836258 5015 scope.go:122] "RemoveContainer" containerID="2b5bdeb6cef9896f1e302c3f182f4c0789720fa4447abdcf526c4e44663da9d7" Nov 03 09:30:18.836671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.836601 5015 scope.go:122] "RemoveContainer" containerID="6c17f7f89c9f58219faa8753ef456cfd3a77b6a65742f268c611b849391230d7" Nov 03 09:30:18.836935 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:18.836906 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm_odh-ai-gateway-infra(5d4aa2d0-e62c-4f37-937e-ab210f4fe52b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" Nov 03 09:30:18.842860 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.842835 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" event={"ID":"ae69616b-adc9-4423-8d8d-5b6b6fb48a81","Type":"ContainerStarted","Data":"da1b9bbbc4e6a422bf489b9ceeeb59d7dcda5a1b820b1b4017e59e0e1b2248c9"} Nov 03 09:30:18.994294 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:18.994262 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm"] Nov 03 09:30:19.072041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:19.071976 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" podStartSLOduration=19.071957945 podStartE2EDuration="19.071957945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:19.070620689 +0000 UTC m=+3051.995562924" watchObservedRunningTime="2026-11-03 09:30:19.071957945 +0000 UTC m=+3051.996900160" Nov 03 09:30:19.205666 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:19.205638 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89525c49-48f6-4822-9a33-159ede94f8af" path="/var/lib/kubelet/pods/89525c49-48f6-4822-9a33-159ede94f8af/volumes" Nov 03 09:30:19.206091 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:19.206073 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" path="/var/lib/kubelet/pods/9e02f4e5-e3f5-4f73-b9ca-62b503719697/volumes" Nov 03 09:30:19.856773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:19.856732 5015 generic.go:356] "Generic (PLEG): container finished" podID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerID="da1b9bbbc4e6a422bf489b9ceeeb59d7dcda5a1b820b1b4017e59e0e1b2248c9" exitCode=60 Nov 03 09:30:19.857197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:19.856819 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" event={"ID":"ae69616b-adc9-4423-8d8d-5b6b6fb48a81","Type":"ContainerDied","Data":"da1b9bbbc4e6a422bf489b9ceeeb59d7dcda5a1b820b1b4017e59e0e1b2248c9"} Nov 03 09:30:19.857197 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:19.856905 5015 scope.go:122] "RemoveContainer" containerID="0d52354f09ee5973480c6d02c7db1286eb2a0f510867bed80a294274eb6560d1" Nov 03 09:30:19.857400 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:19.857373 5015 scope.go:122] "RemoveContainer" containerID="da1b9bbbc4e6a422bf489b9ceeeb59d7dcda5a1b820b1b4017e59e0e1b2248c9" Nov 03 09:30:19.857819 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:19.857794 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5_odh-ai-gateway-infra(ae69616b-adc9-4423-8d8d-5b6b6fb48a81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" Nov 03 09:30:20.021693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:20.021667 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" Nov 03 09:30:20.076746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:20.076665 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-maas-api-service-ca\") pod \"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b\" (UID: \"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b\") " Nov 03 09:30:20.076930 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:20.076844 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-kube-api-access-d5zkl\" (UniqueName: \"kubernetes.io/projected/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-kube-api-access-d5zkl\") pod \"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b\" (UID: \"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b\") " Nov 03 09:30:20.077117 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:20.077092 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-maas-api-service-ca" pod "5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" (UID: "5d4aa2d0-e62c-4f37-937e-ab210f4fe52b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:30:20.077434 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:20.077409 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:20.079595 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:20.079445 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-kube-api-access-d5zkl" pod "5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" (UID: "5d4aa2d0-e62c-4f37-937e-ab210f4fe52b"). InnerVolumeSpecName "kube-api-access-d5zkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:20.086537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:20.086512 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5"] Nov 03 09:30:20.178952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:20.178913 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5zkl\" (UniqueName: \"kubernetes.io/projected/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b-kube-api-access-d5zkl\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:20.866940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:20.866907 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" Nov 03 09:30:20.866940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:20.866928 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm" event={"ID":"5d4aa2d0-e62c-4f37-937e-ab210f4fe52b","Type":"ContainerDied","Data":"9a3d1f5ff2f48834ccc6ea5f0f5d6e34d34b1397e0dee58795d6c3dd1c4eb0d3"} Nov 03 09:30:20.867548 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:20.866995 5015 scope.go:122] "RemoveContainer" containerID="6c17f7f89c9f58219faa8753ef456cfd3a77b6a65742f268c611b849391230d7" Nov 03 09:30:21.113919 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.113890 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" Nov 03 09:30:21.216940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.216854 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-maas-api-service-ca\") pod \"ae69616b-adc9-4423-8d8d-5b6b6fb48a81\" (UID: \"ae69616b-adc9-4423-8d8d-5b6b6fb48a81\") " Nov 03 09:30:21.217120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.217073 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-kube-api-access-5h9gf\" (UniqueName: \"kubernetes.io/projected/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-kube-api-access-5h9gf\") pod \"ae69616b-adc9-4423-8d8d-5b6b6fb48a81\" (UID: \"ae69616b-adc9-4423-8d8d-5b6b6fb48a81\") " Nov 03 09:30:21.217214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.217192 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-maas-api-service-ca" pod "ae69616b-adc9-4423-8d8d-5b6b6fb48a81" (UID: "ae69616b-adc9-4423-8d8d-5b6b6fb48a81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:30:21.217755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.217722 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:21.219752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.219727 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-kube-api-access-5h9gf" pod "ae69616b-adc9-4423-8d8d-5b6b6fb48a81" (UID: "ae69616b-adc9-4423-8d8d-5b6b6fb48a81"). InnerVolumeSpecName "kube-api-access-5h9gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:21.219952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.219937 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm"] Nov 03 09:30:21.223350 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.223326 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894970-jg7lm"] Nov 03 09:30:21.319529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.319502 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h9gf\" (UniqueName: \"kubernetes.io/projected/ae69616b-adc9-4423-8d8d-5b6b6fb48a81-kube-api-access-5h9gf\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:21.880442 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.880408 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" Nov 03 09:30:21.880442 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.880427 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5" event={"ID":"ae69616b-adc9-4423-8d8d-5b6b6fb48a81","Type":"ContainerDied","Data":"16dce8deb13f4ba863e8142351c4623cdf2c10a8ab38908506dd791804b25458"} Nov 03 09:30:21.880962 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:21.880518 5015 scope.go:122] "RemoveContainer" containerID="da1b9bbbc4e6a422bf489b9ceeeb59d7dcda5a1b820b1b4017e59e0e1b2248c9" Nov 03 09:30:22.052370 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:22.052335 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5"] Nov 03 09:30:22.057827 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:22.057790 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894970-z2ww5"] Nov 03 09:30:23.206113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:23.206082 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" path="/var/lib/kubelet/pods/5d4aa2d0-e62c-4f37-937e-ab210f4fe52b/volumes" Nov 03 09:30:23.206545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:23.206522 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" path="/var/lib/kubelet/pods/ae69616b-adc9-4423-8d8d-5b6b6fb48a81/volumes" Nov 03 09:30:28.846418 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.846318 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:30:28.846921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.846800 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:30:28.991166 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.991124 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj"] Nov 03 09:30:28.992124 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992103 5015 cpu_manager.go:414] "containerMap: removing container" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerName="cleanup" Nov 03 09:30:28.992194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992130 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerName="cleanup" Nov 03 09:30:28.992194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992157 5015 cpu_manager.go:414] "containerMap: removing container" podUID="89525c49-48f6-4822-9a33-159ede94f8af" containerName="main" Nov 03 09:30:28.992194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992165 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89525c49-48f6-4822-9a33-159ede94f8af" containerName="main" Nov 03 09:30:28.992194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992177 5015 cpu_manager.go:414] "containerMap: removing container" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerName="cleanup" Nov 03 09:30:28.992194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992185 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerName="cleanup" Nov 03 09:30:28.992356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992201 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerName="cleanup" Nov 03 09:30:28.992356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992208 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerName="cleanup" Nov 03 09:30:28.992356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992228 5015 cpu_manager.go:414] "containerMap: removing container" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerName="cleanup" Nov 03 09:30:28.992356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992235 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerName="cleanup" Nov 03 09:30:28.992356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992252 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerName="cleanup" Nov 03 09:30:28.992356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992259 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerName="cleanup" Nov 03 09:30:28.992356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992287 5015 cpu_manager.go:414] "containerMap: removing container" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerName="cleanup" Nov 03 09:30:28.992356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992293 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerName="cleanup" Nov 03 09:30:28.992615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992465 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerName="cleanup" Nov 03 09:30:28.992615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992499 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerName="cleanup" Nov 03 09:30:28.992615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992514 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerName="cleanup" Nov 03 09:30:28.992615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992525 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerName="cleanup" Nov 03 09:30:28.992615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992540 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerName="cleanup" Nov 03 09:30:28.992615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992554 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="89525c49-48f6-4822-9a33-159ede94f8af" containerName="main" Nov 03 09:30:28.992615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992564 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerName="cleanup" Nov 03 09:30:28.992615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992573 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerName="cleanup" Nov 03 09:30:28.992615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.992581 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerName="cleanup" Nov 03 09:30:28.997809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:28.997788 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:30:29.052150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.052123 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-739a7f\"" Nov 03 09:30:29.080434 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.080399 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj"] Nov 03 09:30:29.114514 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.114412 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615710ea-6ac4-48d5-a36d-4992c64bdf7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj\" (UID: \"615710ea-6ac4-48d5-a36d-4992c64bdf7e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:30:29.114722 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.114615 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h79x7\" (UniqueName: \"kubernetes.io/projected/615710ea-6ac4-48d5-a36d-4992c64bdf7e-kube-api-access-h79x7\") pod \"payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj\" (UID: \"615710ea-6ac4-48d5-a36d-4992c64bdf7e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:30:29.126537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.126506 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw"] Nov 03 09:30:29.127154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.127134 5015 cpu_manager.go:414] "containerMap: removing container" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerName="cleanup" Nov 03 09:30:29.127154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.127151 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e02f4e5-e3f5-4f73-b9ca-62b503719697" containerName="cleanup" Nov 03 09:30:29.127340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.127166 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerName="cleanup" Nov 03 09:30:29.127340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.127171 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerName="cleanup" Nov 03 09:30:29.127340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.127187 5015 cpu_manager.go:414] "containerMap: removing container" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerName="cleanup" Nov 03 09:30:29.127340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.127192 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae69616b-adc9-4423-8d8d-5b6b6fb48a81" containerName="cleanup" Nov 03 09:30:29.127340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.127305 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4aa2d0-e62c-4f37-937e-ab210f4fe52b" containerName="cleanup" Nov 03 09:30:29.131027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.131007 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:30:29.186784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.186759 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-739a7f-dockercfg-zsnhn\"" Nov 03 09:30:29.191833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.191805 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw"] Nov 03 09:30:29.231150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.230927 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h79x7\" (UniqueName: \"kubernetes.io/projected/615710ea-6ac4-48d5-a36d-4992c64bdf7e-kube-api-access-h79x7\") pod \"payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj\" (UID: \"615710ea-6ac4-48d5-a36d-4992c64bdf7e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:30:29.231829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.231803 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vqhx\" (UniqueName: \"kubernetes.io/projected/013d98c5-61e4-4e09-9d86-010d5b9853ce-kube-api-access-9vqhx\") pod \"payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw\" (UID: \"013d98c5-61e4-4e09-9d86-010d5b9853ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:30:29.232137 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.232120 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013d98c5-61e4-4e09-9d86-010d5b9853ce-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw\" (UID: \"013d98c5-61e4-4e09-9d86-010d5b9853ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:30:29.232361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.232345 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615710ea-6ac4-48d5-a36d-4992c64bdf7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj\" (UID: \"615710ea-6ac4-48d5-a36d-4992c64bdf7e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:30:29.233361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.233247 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615710ea-6ac4-48d5-a36d-4992c64bdf7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj\" (UID: \"615710ea-6ac4-48d5-a36d-4992c64bdf7e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:30:29.291182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.291154 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h79x7\" (UniqueName: \"kubernetes.io/projected/615710ea-6ac4-48d5-a36d-4992c64bdf7e-kube-api-access-h79x7\") pod \"payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj\" (UID: \"615710ea-6ac4-48d5-a36d-4992c64bdf7e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:30:29.314515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.314468 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:30:29.334188 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.334159 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqhx\" (UniqueName: \"kubernetes.io/projected/013d98c5-61e4-4e09-9d86-010d5b9853ce-kube-api-access-9vqhx\") pod \"payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw\" (UID: \"013d98c5-61e4-4e09-9d86-010d5b9853ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:30:29.334325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.334236 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013d98c5-61e4-4e09-9d86-010d5b9853ce-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw\" (UID: \"013d98c5-61e4-4e09-9d86-010d5b9853ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:30:29.334998 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.334973 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013d98c5-61e4-4e09-9d86-010d5b9853ce-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw\" (UID: \"013d98c5-61e4-4e09-9d86-010d5b9853ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:30:29.402382 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.402318 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vqhx\" (UniqueName: \"kubernetes.io/projected/013d98c5-61e4-4e09-9d86-010d5b9853ce-kube-api-access-9vqhx\") pod \"payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw\" (UID: \"013d98c5-61e4-4e09-9d86-010d5b9853ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:30:29.448095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.448062 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:30:29.700639 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:30:29.700607 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615710ea_6ac4_48d5_a36d_4992c64bdf7e.slice/crio-19085f1a619d8e62b2492348a4bde6182d57057e8c9ef682efce42dc16014e3a WatchSource:0}: Error finding container 19085f1a619d8e62b2492348a4bde6182d57057e8c9ef682efce42dc16014e3a: Status 404 returned error can't find the container with id 19085f1a619d8e62b2492348a4bde6182d57057e8c9ef682efce42dc16014e3a Nov 03 09:30:29.700761 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.700676 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj"] Nov 03 09:30:29.834932 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:30:29.834901 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013d98c5_61e4_4e09_9d86_010d5b9853ce.slice/crio-dbacf488079364525da38ee4aceac8513f1e70d269b850f46253f5c49cc9f597 WatchSource:0}: Error finding container dbacf488079364525da38ee4aceac8513f1e70d269b850f46253f5c49cc9f597: Status 404 returned error can't find the container with id dbacf488079364525da38ee4aceac8513f1e70d269b850f46253f5c49cc9f597 Nov 03 09:30:29.836989 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.836961 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw"] Nov 03 09:30:29.963602 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.963511 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" event={"ID":"615710ea-6ac4-48d5-a36d-4992c64bdf7e","Type":"ContainerStarted","Data":"343bed9876126995401e9c9cbb31870b617ec53855799ed28d10358e55f5fe1b"} Nov 03 09:30:29.963602 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.963583 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" event={"ID":"615710ea-6ac4-48d5-a36d-4992c64bdf7e","Type":"ContainerStarted","Data":"19085f1a619d8e62b2492348a4bde6182d57057e8c9ef682efce42dc16014e3a"} Nov 03 09:30:29.964130 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.963744 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:30:29.965799 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.965770 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" event={"ID":"013d98c5-61e4-4e09-9d86-010d5b9853ce","Type":"ContainerStarted","Data":"ad9adebb59a7f04579bbcb4d860a6ec261d8cca7803ef77a5766c56df15fde63"} Nov 03 09:30:29.965992 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.965819 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" event={"ID":"013d98c5-61e4-4e09-9d86-010d5b9853ce","Type":"ContainerStarted","Data":"dbacf488079364525da38ee4aceac8513f1e70d269b850f46253f5c49cc9f597"} Nov 03 09:30:29.965992 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:29.965944 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:30:30.199354 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:30.199289 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" podStartSLOduration=2.199272787 podStartE2EDuration="2.199272787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:30.081668469 +0000 UTC m=+3063.006610695" watchObservedRunningTime="2026-11-03 09:30:30.199272787 +0000 UTC m=+3063.124215005" Nov 03 09:30:30.200855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:30.200816 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" podStartSLOduration=1.2008058560000001 podStartE2EDuration="1.200805856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:30.197083298 +0000 UTC m=+3063.122025542" watchObservedRunningTime="2026-11-03 09:30:30.200805856 +0000 UTC m=+3063.125748121" Nov 03 09:30:40.056974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.056943 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b"] Nov 03 09:30:40.062651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.062631 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:30:40.068937 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.068910 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a21e2f0-df22-4854-9b2a-513adf3318ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b\" (UID: \"0a21e2f0-df22-4854-9b2a-513adf3318ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:30:40.069070 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.068961 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgrnt\" (UniqueName: \"kubernetes.io/projected/0a21e2f0-df22-4854-9b2a-513adf3318ca-kube-api-access-wgrnt\") pod \"payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b\" (UID: \"0a21e2f0-df22-4854-9b2a-513adf3318ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:30:40.119510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.119444 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b"] Nov 03 09:30:40.164670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.164635 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f66c6c\"" Nov 03 09:30:40.165182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.165152 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f66c6c-dockercfg-rn74j\"" Nov 03 09:30:40.172135 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.172097 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgrnt\" (UniqueName: \"kubernetes.io/projected/0a21e2f0-df22-4854-9b2a-513adf3318ca-kube-api-access-wgrnt\") pod \"payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b\" (UID: \"0a21e2f0-df22-4854-9b2a-513adf3318ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:30:40.172493 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.172455 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a21e2f0-df22-4854-9b2a-513adf3318ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b\" (UID: \"0a21e2f0-df22-4854-9b2a-513adf3318ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:30:40.174444 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.173849 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a21e2f0-df22-4854-9b2a-513adf3318ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b\" (UID: \"0a21e2f0-df22-4854-9b2a-513adf3318ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:30:40.222307 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.222278 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk"] Nov 03 09:30:40.226767 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.226747 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:30:40.233282 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.233259 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgrnt\" (UniqueName: \"kubernetes.io/projected/0a21e2f0-df22-4854-9b2a-513adf3318ca-kube-api-access-wgrnt\") pod \"payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b\" (UID: \"0a21e2f0-df22-4854-9b2a-513adf3318ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:30:40.269558 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.269529 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk"] Nov 03 09:30:40.273795 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.273764 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk\" (UID: \"6bbff8a4-98eb-4826-a837-ec6c7ae7d458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:30:40.274013 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.273987 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8rgj\" (UniqueName: \"kubernetes.io/projected/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-kube-api-access-d8rgj\") pod \"payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk\" (UID: \"6bbff8a4-98eb-4826-a837-ec6c7ae7d458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:30:40.376345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.376270 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk\" (UID: \"6bbff8a4-98eb-4826-a837-ec6c7ae7d458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:30:40.376538 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.376367 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8rgj\" (UniqueName: \"kubernetes.io/projected/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-kube-api-access-d8rgj\") pod \"payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk\" (UID: \"6bbff8a4-98eb-4826-a837-ec6c7ae7d458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:30:40.376979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.376957 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk\" (UID: \"6bbff8a4-98eb-4826-a837-ec6c7ae7d458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:30:40.378177 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.378156 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:30:40.436425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.436389 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8rgj\" (UniqueName: \"kubernetes.io/projected/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-kube-api-access-d8rgj\") pod \"payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk\" (UID: \"6bbff8a4-98eb-4826-a837-ec6c7ae7d458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:30:40.554078 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.554050 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:30:40.760228 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:30:40.760197 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a21e2f0_df22_4854_9b2a_513adf3318ca.slice/crio-16f17335d47dff957ea18cdb8883e2796cc0cdbdd65c1ff0ae727a62b93ed9e1 WatchSource:0}: Error finding container 16f17335d47dff957ea18cdb8883e2796cc0cdbdd65c1ff0ae727a62b93ed9e1: Status 404 returned error can't find the container with id 16f17335d47dff957ea18cdb8883e2796cc0cdbdd65c1ff0ae727a62b93ed9e1 Nov 03 09:30:40.760762 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.760737 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b"] Nov 03 09:30:40.935377 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.935344 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk"] Nov 03 09:30:40.936153 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:30:40.936125 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbff8a4_98eb_4826_a837_ec6c7ae7d458.slice/crio-668f58164ce34d265fe6379f30726a2954e3b3fd293b2adb84661e1c3e23d866 WatchSource:0}: Error finding container 668f58164ce34d265fe6379f30726a2954e3b3fd293b2adb84661e1c3e23d866: Status 404 returned error can't find the container with id 668f58164ce34d265fe6379f30726a2954e3b3fd293b2adb84661e1c3e23d866 Nov 03 09:30:40.979097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.979064 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:30:40.979231 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:40.979205 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:30:41.051184 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:41.051150 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" event={"ID":"0a21e2f0-df22-4854-9b2a-513adf3318ca","Type":"ContainerStarted","Data":"1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608"} Nov 03 09:30:41.052232 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:41.051215 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" event={"ID":"0a21e2f0-df22-4854-9b2a-513adf3318ca","Type":"ContainerStarted","Data":"16f17335d47dff957ea18cdb8883e2796cc0cdbdd65c1ff0ae727a62b93ed9e1"} Nov 03 09:30:41.052232 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:41.051346 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:30:41.054011 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:41.053984 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" event={"ID":"6bbff8a4-98eb-4826-a837-ec6c7ae7d458","Type":"ContainerStarted","Data":"f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f"} Nov 03 09:30:41.054131 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:41.054038 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" event={"ID":"6bbff8a4-98eb-4826-a837-ec6c7ae7d458","Type":"ContainerStarted","Data":"668f58164ce34d265fe6379f30726a2954e3b3fd293b2adb84661e1c3e23d866"} Nov 03 09:30:41.054199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:41.054150 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:30:41.340222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:41.340086 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" podStartSLOduration=1.340067123 podStartE2EDuration="1.340067123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:41.333323762 +0000 UTC m=+3074.258265980" watchObservedRunningTime="2026-11-03 09:30:41.340067123 +0000 UTC m=+3074.265009348" Nov 03 09:30:41.470673 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:41.469923 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" podStartSLOduration=1.469903832 podStartE2EDuration="1.469903832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:41.46929864 +0000 UTC m=+3074.394240877" watchObservedRunningTime="2026-11-03 09:30:41.469903832 +0000 UTC m=+3074.394846050" Nov 03 09:30:46.902866 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:30:46.902829 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:30:47.643025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.642992 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw"] Nov 03 09:30:47.648701 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.648681 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:30:47.756836 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.756794 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f01a6d-dockercfg-rwxgz\"" Nov 03 09:30:47.756836 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.756794 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f01a6d\"" Nov 03 09:30:47.761133 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.761103 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw"] Nov 03 09:30:47.769966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.769931 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkqg\" (UniqueName: \"kubernetes.io/projected/b0932073-b373-4c45-9ae6-76f6c9cab5e0-kube-api-access-zkkqg\") pod \"payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw\" (UID: \"b0932073-b373-4c45-9ae6-76f6c9cab5e0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:30:47.770093 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.769983 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0932073-b373-4c45-9ae6-76f6c9cab5e0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw\" (UID: \"b0932073-b373-4c45-9ae6-76f6c9cab5e0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:30:47.871886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.871849 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkqg\" (UniqueName: \"kubernetes.io/projected/b0932073-b373-4c45-9ae6-76f6c9cab5e0-kube-api-access-zkkqg\") pod \"payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw\" (UID: \"b0932073-b373-4c45-9ae6-76f6c9cab5e0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:30:47.872069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.871901 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0932073-b373-4c45-9ae6-76f6c9cab5e0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw\" (UID: \"b0932073-b373-4c45-9ae6-76f6c9cab5e0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:30:47.872679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.872658 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0932073-b373-4c45-9ae6-76f6c9cab5e0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw\" (UID: \"b0932073-b373-4c45-9ae6-76f6c9cab5e0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:30:47.932091 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.932018 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkqg\" (UniqueName: \"kubernetes.io/projected/b0932073-b373-4c45-9ae6-76f6c9cab5e0-kube-api-access-zkkqg\") pod \"payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw\" (UID: \"b0932073-b373-4c45-9ae6-76f6c9cab5e0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:30:47.963813 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:47.963780 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:30:48.349342 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:30:48.349315 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0932073_b373_4c45_9ae6_76f6c9cab5e0.slice/crio-7e77a150c08ea4477aca0b70e15d699b0aa30770c977442ef5cff251ac1e9516 WatchSource:0}: Error finding container 7e77a150c08ea4477aca0b70e15d699b0aa30770c977442ef5cff251ac1e9516: Status 404 returned error can't find the container with id 7e77a150c08ea4477aca0b70e15d699b0aa30770c977442ef5cff251ac1e9516 Nov 03 09:30:48.349609 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:48.349585 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw"] Nov 03 09:30:49.129078 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:49.129024 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" event={"ID":"b0932073-b373-4c45-9ae6-76f6c9cab5e0","Type":"ContainerStarted","Data":"6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e"} Nov 03 09:30:49.129598 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:49.129103 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" event={"ID":"b0932073-b373-4c45-9ae6-76f6c9cab5e0","Type":"ContainerStarted","Data":"7e77a150c08ea4477aca0b70e15d699b0aa30770c977442ef5cff251ac1e9516"} Nov 03 09:30:49.129598 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:49.129135 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:30:49.247121 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:49.247067 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" podStartSLOduration=2.247053611 podStartE2EDuration="2.247053611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:49.244535389 +0000 UTC m=+3082.169477606" watchObservedRunningTime="2026-11-03 09:30:49.247053611 +0000 UTC m=+3082.171995828" Nov 03 09:30:52.064335 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:52.064301 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:30:52.064886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:30:52.064388 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:31:00.139949 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:00.139919 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:31:16.903364 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:31:16.903283 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:31:46.903067 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:31:46.903031 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:31:59.333524 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.333454 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd"] Nov 03 09:31:59.338055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.338032 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:31:59.398648 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.398611 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd"] Nov 03 09:31:59.441738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.441707 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-f66c6c-serving-cert\"" Nov 03 09:31:59.455598 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.455567 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzrh\" (UniqueName: \"kubernetes.io/projected/caaf72c6-18eb-4ec4-ac02-43f5d4095027-kube-api-access-bkzrh\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:31:59.455775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.455601 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caaf72c6-18eb-4ec4-ac02-43f5d4095027-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:31:59.455775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.455749 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:31:59.455876 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.455782 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-metrics-certs\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:31:59.557304 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.557270 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-metrics-certs\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:31:59.557539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.557367 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzrh\" (UniqueName: \"kubernetes.io/projected/caaf72c6-18eb-4ec4-ac02-43f5d4095027-kube-api-access-bkzrh\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:31:59.557539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.557393 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caaf72c6-18eb-4ec4-ac02-43f5d4095027-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:31:59.557539 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.557514 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:31:59.557731 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:31:59.557616 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-serving-cert: secret "maas-api-e2e-extmod-a-f66c6c-serving-cert" not found Nov 03 09:31:59.557731 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:31:59.557669 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-maas-api-tls podName:caaf72c6-18eb-4ec4-ac02-43f5d4095027 nodeName:}" failed. No retries permitted until 2026-11-03 09:32:00.057654435 +0000 UTC m=+3152.982596630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-maas-api-tls") pod "maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" (UID: "caaf72c6-18eb-4ec4-ac02-43f5d4095027") : secret "maas-api-e2e-extmod-a-f66c6c-serving-cert" not found Nov 03 09:31:59.558206 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.558178 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caaf72c6-18eb-4ec4-ac02-43f5d4095027-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:31:59.560018 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.560000 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-metrics-certs\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:31:59.620181 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:31:59.620106 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzrh\" (UniqueName: \"kubernetes.io/projected/caaf72c6-18eb-4ec4-ac02-43f5d4095027-kube-api-access-bkzrh\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:32:00.063382 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:32:00.063349 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:32:00.066091 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:32:00.066071 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:32:00.254872 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:32:00.254835 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:32:00.660250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:32:00.660217 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd"] Nov 03 09:32:00.660888 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:32:00.660847 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaaf72c6_18eb_4ec4_ac02_43f5d4095027.slice/crio-6d165d090bbf3ab1d29d5a5e5ecfc8c5f02019a38fc2dd161edc6252a5eb203d WatchSource:0}: Error finding container 6d165d090bbf3ab1d29d5a5e5ecfc8c5f02019a38fc2dd161edc6252a5eb203d: Status 404 returned error can't find the container with id 6d165d090bbf3ab1d29d5a5e5ecfc8c5f02019a38fc2dd161edc6252a5eb203d Nov 03 09:32:00.662436 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:32:00.662415 5015 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:32:00.722660 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:32:00.722625 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" event={"ID":"caaf72c6-18eb-4ec4-ac02-43f5d4095027","Type":"ContainerStarted","Data":"6d165d090bbf3ab1d29d5a5e5ecfc8c5f02019a38fc2dd161edc6252a5eb203d"} Nov 03 09:32:01.736346 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:32:01.736247 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" event={"ID":"caaf72c6-18eb-4ec4-ac02-43f5d4095027","Type":"ContainerStarted","Data":"f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074"} Nov 03 09:32:01.736822 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:32:01.736382 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:32:01.861995 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:32:01.861938 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" podStartSLOduration=2.038302784 podStartE2EDuration="2.861923295s" totalImagesPullingTime="823.620511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:32:01.859632996 +0000 UTC m=+3154.784575237" watchObservedRunningTime="2026-11-03 09:32:01.861923295 +0000 UTC m=+3154.786865511" Nov 03 09:32:07.753354 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:32:07.753324 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:32:16.903077 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:32:16.903028 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:32:46.907779 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:32:46.907689 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:33:16.904798 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:33:16.904754 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:33:17.282320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.282284 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr"] Nov 03 09:33:17.286602 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.286585 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:33:17.348429 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.348379 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr"] Nov 03 09:33:17.361666 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.361620 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00af5a6c-af44-4a21-b197-c25b51ece5df-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr\" (UID: \"00af5a6c-af44-4a21-b197-c25b51ece5df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:33:17.361912 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.361707 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc2x7\" (UniqueName: \"kubernetes.io/projected/00af5a6c-af44-4a21-b197-c25b51ece5df-kube-api-access-gc2x7\") pod \"payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr\" (UID: \"00af5a6c-af44-4a21-b197-c25b51ece5df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:33:17.391130 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.391103 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-5adb63\"" Nov 03 09:33:17.391278 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.391137 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-5adb63-dockercfg-stzfl\"" Nov 03 09:33:17.462459 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.462428 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00af5a6c-af44-4a21-b197-c25b51ece5df-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr\" (UID: \"00af5a6c-af44-4a21-b197-c25b51ece5df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:33:17.462692 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.462556 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2x7\" (UniqueName: \"kubernetes.io/projected/00af5a6c-af44-4a21-b197-c25b51ece5df-kube-api-access-gc2x7\") pod \"payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr\" (UID: \"00af5a6c-af44-4a21-b197-c25b51ece5df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:33:17.463146 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.463126 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00af5a6c-af44-4a21-b197-c25b51ece5df-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr\" (UID: \"00af5a6c-af44-4a21-b197-c25b51ece5df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:33:17.522809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.522779 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2x7\" (UniqueName: \"kubernetes.io/projected/00af5a6c-af44-4a21-b197-c25b51ece5df-kube-api-access-gc2x7\") pod \"payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr\" (UID: \"00af5a6c-af44-4a21-b197-c25b51ece5df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:33:17.601309 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.601229 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:33:17.989357 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:17.989329 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr"] Nov 03 09:33:17.990044 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:33:17.990011 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00af5a6c_af44_4a21_b197_c25b51ece5df.slice/crio-6ea8576514b581855ccf702ae41c115b4320245df5d24bfdc0335396e90c653b WatchSource:0}: Error finding container 6ea8576514b581855ccf702ae41c115b4320245df5d24bfdc0335396e90c653b: Status 404 returned error can't find the container with id 6ea8576514b581855ccf702ae41c115b4320245df5d24bfdc0335396e90c653b Nov 03 09:33:18.360285 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:18.360197 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" event={"ID":"00af5a6c-af44-4a21-b197-c25b51ece5df","Type":"ContainerStarted","Data":"0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb"} Nov 03 09:33:18.360285 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:18.360266 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" event={"ID":"00af5a6c-af44-4a21-b197-c25b51ece5df","Type":"ContainerStarted","Data":"6ea8576514b581855ccf702ae41c115b4320245df5d24bfdc0335396e90c653b"} Nov 03 09:33:18.360507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:18.360301 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:33:18.479145 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:18.479077 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" podStartSLOduration=1.47905814 podStartE2EDuration="1.47905814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:33:18.477797953 +0000 UTC m=+3231.402740170" watchObservedRunningTime="2026-11-03 09:33:18.47905814 +0000 UTC m=+3231.404000356" Nov 03 09:33:29.371778 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:29.371748 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:33:33.293973 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:33.293938 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw"] Nov 03 09:33:33.404789 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:33.404758 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj"] Nov 03 09:33:38.294498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.294436 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" podUID="013d98c5-61e4-4e09-9d86-010d5b9853ce" containerName="payload-processing" containerID="cri-o://ad9adebb59a7f04579bbcb4d860a6ec261d8cca7803ef77a5766c56df15fde63" gracePeriod=25 Nov 03 09:33:38.405815 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.405752 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" podUID="615710ea-6ac4-48d5-a36d-4992c64bdf7e" containerName="payload-pre-processing" containerID="cri-o://343bed9876126995401e9c9cbb31870b617ec53855799ed28d10358e55f5fe1b" gracePeriod=25 Nov 03 09:33:38.525576 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.525546 5015 generic.go:356] "Generic (PLEG): container finished" podID="615710ea-6ac4-48d5-a36d-4992c64bdf7e" containerID="343bed9876126995401e9c9cbb31870b617ec53855799ed28d10358e55f5fe1b" exitCode=0 Nov 03 09:33:38.525694 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.525615 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" event={"ID":"615710ea-6ac4-48d5-a36d-4992c64bdf7e","Type":"ContainerDied","Data":"343bed9876126995401e9c9cbb31870b617ec53855799ed28d10358e55f5fe1b"} Nov 03 09:33:38.527527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.527504 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" event={"ID":"013d98c5-61e4-4e09-9d86-010d5b9853ce","Type":"ContainerDied","Data":"ad9adebb59a7f04579bbcb4d860a6ec261d8cca7803ef77a5766c56df15fde63"} Nov 03 09:33:38.527623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.527504 5015 generic.go:356] "Generic (PLEG): container finished" podID="013d98c5-61e4-4e09-9d86-010d5b9853ce" containerID="ad9adebb59a7f04579bbcb4d860a6ec261d8cca7803ef77a5766c56df15fde63" exitCode=0 Nov 03 09:33:38.589092 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.589065 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:33:38.625559 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.625515 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013d98c5-61e4-4e09-9d86-010d5b9853ce-kube-api-access-9vqhx\" (UniqueName: \"kubernetes.io/projected/013d98c5-61e4-4e09-9d86-010d5b9853ce-kube-api-access-9vqhx\") pod \"013d98c5-61e4-4e09-9d86-010d5b9853ce\" (UID: \"013d98c5-61e4-4e09-9d86-010d5b9853ce\") " Nov 03 09:33:38.625757 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.625676 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013d98c5-61e4-4e09-9d86-010d5b9853ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013d98c5-61e4-4e09-9d86-010d5b9853ce-plugins-config-volume\") pod \"013d98c5-61e4-4e09-9d86-010d5b9853ce\" (UID: \"013d98c5-61e4-4e09-9d86-010d5b9853ce\") " Nov 03 09:33:38.626347 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.626313 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013d98c5-61e4-4e09-9d86-010d5b9853ce-plugins-config-volume" pod "013d98c5-61e4-4e09-9d86-010d5b9853ce" (UID: "013d98c5-61e4-4e09-9d86-010d5b9853ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:33:38.633310 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.628996 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013d98c5-61e4-4e09-9d86-010d5b9853ce-kube-api-access-9vqhx" pod "013d98c5-61e4-4e09-9d86-010d5b9853ce" (UID: "013d98c5-61e4-4e09-9d86-010d5b9853ce"). InnerVolumeSpecName "kube-api-access-9vqhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:33:38.688887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.688860 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:33:38.727402 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.727370 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615710ea-6ac4-48d5-a36d-4992c64bdf7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615710ea-6ac4-48d5-a36d-4992c64bdf7e-plugins-config-volume\") pod \"615710ea-6ac4-48d5-a36d-4992c64bdf7e\" (UID: \"615710ea-6ac4-48d5-a36d-4992c64bdf7e\") " Nov 03 09:33:38.727598 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.727444 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615710ea-6ac4-48d5-a36d-4992c64bdf7e-kube-api-access-h79x7\" (UniqueName: \"kubernetes.io/projected/615710ea-6ac4-48d5-a36d-4992c64bdf7e-kube-api-access-h79x7\") pod \"615710ea-6ac4-48d5-a36d-4992c64bdf7e\" (UID: \"615710ea-6ac4-48d5-a36d-4992c64bdf7e\") " Nov 03 09:33:38.731510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.727849 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615710ea-6ac4-48d5-a36d-4992c64bdf7e-plugins-config-volume" pod "615710ea-6ac4-48d5-a36d-4992c64bdf7e" (UID: "615710ea-6ac4-48d5-a36d-4992c64bdf7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:33:38.731510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.728837 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vqhx\" (UniqueName: \"kubernetes.io/projected/013d98c5-61e4-4e09-9d86-010d5b9853ce-kube-api-access-9vqhx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:33:38.731510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.728878 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013d98c5-61e4-4e09-9d86-010d5b9853ce-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:33:38.745602 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.745565 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615710ea-6ac4-48d5-a36d-4992c64bdf7e-kube-api-access-h79x7" pod "615710ea-6ac4-48d5-a36d-4992c64bdf7e" (UID: "615710ea-6ac4-48d5-a36d-4992c64bdf7e"). InnerVolumeSpecName "kube-api-access-h79x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:33:38.830589 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.830508 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615710ea-6ac4-48d5-a36d-4992c64bdf7e-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:33:38.830589 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:38.830536 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h79x7\" (UniqueName: \"kubernetes.io/projected/615710ea-6ac4-48d5-a36d-4992c64bdf7e-kube-api-access-h79x7\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:33:39.536155 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:39.536121 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" Nov 03 09:33:39.536685 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:39.536119 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj" event={"ID":"615710ea-6ac4-48d5-a36d-4992c64bdf7e","Type":"ContainerDied","Data":"19085f1a619d8e62b2492348a4bde6182d57057e8c9ef682efce42dc16014e3a"} Nov 03 09:33:39.536685 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:39.536278 5015 scope.go:122] "RemoveContainer" containerID="343bed9876126995401e9c9cbb31870b617ec53855799ed28d10358e55f5fe1b" Nov 03 09:33:39.538095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:39.538073 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" Nov 03 09:33:39.538373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:39.538102 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw" event={"ID":"013d98c5-61e4-4e09-9d86-010d5b9853ce","Type":"ContainerDied","Data":"dbacf488079364525da38ee4aceac8513f1e70d269b850f46253f5c49cc9f597"} Nov 03 09:33:39.548046 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:39.548030 5015 scope.go:122] "RemoveContainer" containerID="ad9adebb59a7f04579bbcb4d860a6ec261d8cca7803ef77a5766c56df15fde63" Nov 03 09:33:39.707375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:39.707338 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw"] Nov 03 09:33:39.757916 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:39.757885 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-739a7f-5d88899ccc-666xw"] Nov 03 09:33:39.827579 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:39.827499 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj"] Nov 03 09:33:39.833993 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:39.833965 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-739a7f-79596b9b59-j4lpj"] Nov 03 09:33:41.206049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:41.206004 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013d98c5-61e4-4e09-9d86-010d5b9853ce" path="/var/lib/kubelet/pods/013d98c5-61e4-4e09-9d86-010d5b9853ce/volumes" Nov 03 09:33:41.206663 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:41.206642 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615710ea-6ac4-48d5-a36d-4992c64bdf7e" path="/var/lib/kubelet/pods/615710ea-6ac4-48d5-a36d-4992c64bdf7e/volumes" Nov 03 09:33:46.660614 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:46.660579 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk"] Nov 03 09:33:46.772503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:46.772450 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b"] Nov 03 09:33:46.906092 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:33:46.906055 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:33:49.749460 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:49.749424 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw"] Nov 03 09:33:51.661445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:51.661406 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" podUID="6bbff8a4-98eb-4826-a837-ec6c7ae7d458" containerName="payload-processing" containerID="cri-o://f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f" gracePeriod=25 Nov 03 09:33:51.773818 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:51.773773 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" podUID="0a21e2f0-df22-4854-9b2a-513adf3318ca" containerName="payload-pre-processing" containerID="cri-o://1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608" gracePeriod=25 Nov 03 09:33:51.947675 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:51.947643 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:33:52.069574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.069543 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:33:52.102226 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.102196 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-kube-api-access-d8rgj\" (UniqueName: \"kubernetes.io/projected/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-kube-api-access-d8rgj\") pod \"6bbff8a4-98eb-4826-a837-ec6c7ae7d458\" (UID: \"6bbff8a4-98eb-4826-a837-ec6c7ae7d458\") " Nov 03 09:33:52.102424 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.102354 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-plugins-config-volume\") pod \"6bbff8a4-98eb-4826-a837-ec6c7ae7d458\" (UID: \"6bbff8a4-98eb-4826-a837-ec6c7ae7d458\") " Nov 03 09:33:52.102802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.102763 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-plugins-config-volume" pod "6bbff8a4-98eb-4826-a837-ec6c7ae7d458" (UID: "6bbff8a4-98eb-4826-a837-ec6c7ae7d458"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:33:52.103250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.102891 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:33:52.104707 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.104681 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-kube-api-access-d8rgj" pod "6bbff8a4-98eb-4826-a837-ec6c7ae7d458" (UID: "6bbff8a4-98eb-4826-a837-ec6c7ae7d458"). InnerVolumeSpecName "kube-api-access-d8rgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:33:52.203911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.203829 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a21e2f0-df22-4854-9b2a-513adf3318ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a21e2f0-df22-4854-9b2a-513adf3318ca-plugins-config-volume\") pod \"0a21e2f0-df22-4854-9b2a-513adf3318ca\" (UID: \"0a21e2f0-df22-4854-9b2a-513adf3318ca\") " Nov 03 09:33:52.204063 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.203994 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a21e2f0-df22-4854-9b2a-513adf3318ca-kube-api-access-wgrnt\" (UniqueName: \"kubernetes.io/projected/0a21e2f0-df22-4854-9b2a-513adf3318ca-kube-api-access-wgrnt\") pod \"0a21e2f0-df22-4854-9b2a-513adf3318ca\" (UID: \"0a21e2f0-df22-4854-9b2a-513adf3318ca\") " Nov 03 09:33:52.204205 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.204184 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a21e2f0-df22-4854-9b2a-513adf3318ca-plugins-config-volume" pod "0a21e2f0-df22-4854-9b2a-513adf3318ca" (UID: "0a21e2f0-df22-4854-9b2a-513adf3318ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:33:52.204344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.204331 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8rgj\" (UniqueName: \"kubernetes.io/projected/6bbff8a4-98eb-4826-a837-ec6c7ae7d458-kube-api-access-d8rgj\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:33:52.204385 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.204348 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a21e2f0-df22-4854-9b2a-513adf3318ca-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:33:52.206403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.206381 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a21e2f0-df22-4854-9b2a-513adf3318ca-kube-api-access-wgrnt" pod "0a21e2f0-df22-4854-9b2a-513adf3318ca" (UID: "0a21e2f0-df22-4854-9b2a-513adf3318ca"). InnerVolumeSpecName "kube-api-access-wgrnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:33:52.305516 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.305441 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgrnt\" (UniqueName: \"kubernetes.io/projected/0a21e2f0-df22-4854-9b2a-513adf3318ca-kube-api-access-wgrnt\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:33:52.653547 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.653489 5015 generic.go:356] "Generic (PLEG): container finished" podID="0a21e2f0-df22-4854-9b2a-513adf3318ca" containerID="1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608" exitCode=0 Nov 03 09:33:52.653547 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.653503 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" event={"ID":"0a21e2f0-df22-4854-9b2a-513adf3318ca","Type":"ContainerDied","Data":"1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608"} Nov 03 09:33:52.653824 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.653555 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" Nov 03 09:33:52.653824 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.653622 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" event={"ID":"0a21e2f0-df22-4854-9b2a-513adf3318ca","Type":"ContainerDied","Data":"16f17335d47dff957ea18cdb8883e2796cc0cdbdd65c1ff0ae727a62b93ed9e1"} Nov 03 09:33:52.653824 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.653658 5015 scope.go:122] "RemoveContainer" containerID="1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608" Nov 03 09:33:52.656359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.656328 5015 generic.go:356] "Generic (PLEG): container finished" podID="6bbff8a4-98eb-4826-a837-ec6c7ae7d458" containerID="f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f" exitCode=0 Nov 03 09:33:52.656535 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.656412 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" event={"ID":"6bbff8a4-98eb-4826-a837-ec6c7ae7d458","Type":"ContainerDied","Data":"f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f"} Nov 03 09:33:52.656535 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.656448 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" Nov 03 09:33:52.657201 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.656453 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk" event={"ID":"6bbff8a4-98eb-4826-a837-ec6c7ae7d458","Type":"ContainerDied","Data":"668f58164ce34d265fe6379f30726a2954e3b3fd293b2adb84661e1c3e23d866"} Nov 03 09:33:52.668275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.667998 5015 scope.go:122] "RemoveContainer" containerID="1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608" Nov 03 09:33:52.668870 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:33:52.668376 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608\": container with ID starting with 1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608 not found: ID does not exist" containerID="1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608" Nov 03 09:33:52.668870 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.668411 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608"} err="failed to get container status \"1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608\": rpc error: code = NotFound desc = could not find container \"1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608\": container with ID starting with 1ae01b71040329c089111e64b13dfbabe92643a050944e1c826e9d63722f5608 not found: ID does not exist" Nov 03 09:33:52.668870 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.668436 5015 scope.go:122] "RemoveContainer" containerID="f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f" Nov 03 09:33:52.691316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.691296 5015 scope.go:122] "RemoveContainer" containerID="f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f" Nov 03 09:33:52.691626 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:33:52.691602 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f\": container with ID starting with f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f not found: ID does not exist" containerID="f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f" Nov 03 09:33:52.691702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.691637 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f"} err="failed to get container status \"f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f\": rpc error: code = NotFound desc = could not find container \"f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f\": container with ID starting with f909085e986770b402c816a42045bbe23fef713974eb7966b25c5c2590d1444f not found: ID does not exist" Nov 03 09:33:52.825718 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.825682 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b"] Nov 03 09:33:52.876527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.876465 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b"] Nov 03 09:33:52.945128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.945032 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk"] Nov 03 09:33:52.951046 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:52.951007 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f66c6c-8474495d9f-g72rk"] Nov 03 09:33:53.063061 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:53.063012 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f66c6c-6c7766f45f-v9b7b" podUID="0a21e2f0-df22-4854-9b2a-513adf3318ca" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.209:50052\" within 1s: context deadline exceeded" Nov 03 09:33:53.063272 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:33:53.063100 5015 logging.go:55] [core] [Channel #10176 SubChannel #10177]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.209:50052", ServerName: "10.132.0.209:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.209:50052: operation was canceled" Nov 03 09:33:53.206981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:53.206904 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a21e2f0-df22-4854-9b2a-513adf3318ca" path="/var/lib/kubelet/pods/0a21e2f0-df22-4854-9b2a-513adf3318ca/volumes" Nov 03 09:33:53.207352 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:53.207338 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bbff8a4-98eb-4826-a837-ec6c7ae7d458" path="/var/lib/kubelet/pods/6bbff8a4-98eb-4826-a837-ec6c7ae7d458/volumes" Nov 03 09:33:54.750023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:54.749982 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" podUID="b0932073-b373-4c45-9ae6-76f6c9cab5e0" containerName="payload-processing" containerID="cri-o://6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e" gracePeriod=25 Nov 03 09:33:55.028124 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.028097 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:33:55.034885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.034865 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0932073-b373-4c45-9ae6-76f6c9cab5e0-kube-api-access-zkkqg\" (UniqueName: \"kubernetes.io/projected/b0932073-b373-4c45-9ae6-76f6c9cab5e0-kube-api-access-zkkqg\") pod \"b0932073-b373-4c45-9ae6-76f6c9cab5e0\" (UID: \"b0932073-b373-4c45-9ae6-76f6c9cab5e0\") " Nov 03 09:33:55.034972 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.034927 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0932073-b373-4c45-9ae6-76f6c9cab5e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0932073-b373-4c45-9ae6-76f6c9cab5e0-plugins-config-volume\") pod \"b0932073-b373-4c45-9ae6-76f6c9cab5e0\" (UID: \"b0932073-b373-4c45-9ae6-76f6c9cab5e0\") " Nov 03 09:33:55.035341 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.035317 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0932073-b373-4c45-9ae6-76f6c9cab5e0-plugins-config-volume" pod "b0932073-b373-4c45-9ae6-76f6c9cab5e0" (UID: "b0932073-b373-4c45-9ae6-76f6c9cab5e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:33:55.035497 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.035464 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0932073-b373-4c45-9ae6-76f6c9cab5e0-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:33:55.037154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.037126 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0932073-b373-4c45-9ae6-76f6c9cab5e0-kube-api-access-zkkqg" pod "b0932073-b373-4c45-9ae6-76f6c9cab5e0" (UID: "b0932073-b373-4c45-9ae6-76f6c9cab5e0"). InnerVolumeSpecName "kube-api-access-zkkqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:33:55.136398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.136366 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkkqg\" (UniqueName: \"kubernetes.io/projected/b0932073-b373-4c45-9ae6-76f6c9cab5e0-kube-api-access-zkkqg\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:33:55.690110 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.690076 5015 generic.go:356] "Generic (PLEG): container finished" podID="b0932073-b373-4c45-9ae6-76f6c9cab5e0" containerID="6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e" exitCode=0 Nov 03 09:33:55.690320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.690143 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" Nov 03 09:33:55.690320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.690156 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" event={"ID":"b0932073-b373-4c45-9ae6-76f6c9cab5e0","Type":"ContainerDied","Data":"6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e"} Nov 03 09:33:55.690320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.690224 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw" event={"ID":"b0932073-b373-4c45-9ae6-76f6c9cab5e0","Type":"ContainerDied","Data":"7e77a150c08ea4477aca0b70e15d699b0aa30770c977442ef5cff251ac1e9516"} Nov 03 09:33:55.690320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.690257 5015 scope.go:122] "RemoveContainer" containerID="6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e" Nov 03 09:33:55.700905 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.700883 5015 scope.go:122] "RemoveContainer" containerID="6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e" Nov 03 09:33:55.701250 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:33:55.701227 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e\": container with ID starting with 6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e not found: ID does not exist" containerID="6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e" Nov 03 09:33:55.701347 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.701258 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e"} err="failed to get container status \"6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e\": rpc error: code = NotFound desc = could not find container \"6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e\": container with ID starting with 6cd6ccf9421f12081ee2127884df0b5a3f05188decfbc1935a77370427ffe49e not found: ID does not exist" Nov 03 09:33:55.863078 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.863042 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw"] Nov 03 09:33:55.911035 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:55.911004 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f01a6d-6bc58d4f79-wl7kw"] Nov 03 09:33:57.207308 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:33:57.207280 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0932073-b373-4c45-9ae6-76f6c9cab5e0" path="/var/lib/kubelet/pods/b0932073-b373-4c45-9ae6-76f6c9cab5e0/volumes" Nov 03 09:34:16.903709 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:34:16.903623 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:34:27.829173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:27.829142 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:34:27.845970 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:27.845948 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:34:46.903268 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:34:46.903225 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:34:49.173910 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.173867 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww"] Nov 03 09:34:49.174556 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174541 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0a21e2f0-df22-4854-9b2a-513adf3318ca" containerName="payload-pre-processing" Nov 03 09:34:49.174615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174557 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a21e2f0-df22-4854-9b2a-513adf3318ca" containerName="payload-pre-processing" Nov 03 09:34:49.174615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174568 5015 cpu_manager.go:414] "containerMap: removing container" podUID="b0932073-b373-4c45-9ae6-76f6c9cab5e0" containerName="payload-processing" Nov 03 09:34:49.174615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174573 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0932073-b373-4c45-9ae6-76f6c9cab5e0" containerName="payload-processing" Nov 03 09:34:49.174615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174597 5015 cpu_manager.go:414] "containerMap: removing container" podUID="615710ea-6ac4-48d5-a36d-4992c64bdf7e" containerName="payload-pre-processing" Nov 03 09:34:49.174615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174602 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615710ea-6ac4-48d5-a36d-4992c64bdf7e" containerName="payload-pre-processing" Nov 03 09:34:49.174615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174613 5015 cpu_manager.go:414] "containerMap: removing container" podUID="013d98c5-61e4-4e09-9d86-010d5b9853ce" containerName="payload-processing" Nov 03 09:34:49.174615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174618 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013d98c5-61e4-4e09-9d86-010d5b9853ce" containerName="payload-processing" Nov 03 09:34:49.174847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174624 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6bbff8a4-98eb-4826-a837-ec6c7ae7d458" containerName="payload-processing" Nov 03 09:34:49.174847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174629 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbff8a4-98eb-4826-a837-ec6c7ae7d458" containerName="payload-processing" Nov 03 09:34:49.174847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174739 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="013d98c5-61e4-4e09-9d86-010d5b9853ce" containerName="payload-processing" Nov 03 09:34:49.174847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174749 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbff8a4-98eb-4826-a837-ec6c7ae7d458" containerName="payload-processing" Nov 03 09:34:49.174847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174760 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a21e2f0-df22-4854-9b2a-513adf3318ca" containerName="payload-pre-processing" Nov 03 09:34:49.174847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174767 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0932073-b373-4c45-9ae6-76f6c9cab5e0" containerName="payload-processing" Nov 03 09:34:49.174847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.174777 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="615710ea-6ac4-48d5-a36d-4992c64bdf7e" containerName="payload-pre-processing" Nov 03 09:34:49.180027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.180012 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.236581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.236549 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww"] Nov 03 09:34:49.282218 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.282186 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-5adb63-serving-cert\"" Nov 03 09:34:49.287827 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.287801 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4bzz\" (UniqueName: \"kubernetes.io/projected/74e77e1c-5e86-424c-bf70-6d34265486f9-kube-api-access-q4bzz\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.287924 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.287879 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.287975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.287942 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.288054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.288041 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e77e1c-5e86-424c-bf70-6d34265486f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.389064 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.389031 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e77e1c-5e86-424c-bf70-6d34265486f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.389254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.389117 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4bzz\" (UniqueName: \"kubernetes.io/projected/74e77e1c-5e86-424c-bf70-6d34265486f9-kube-api-access-q4bzz\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.389254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.389163 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.389254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.389209 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.389378 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:34:49.389337 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-serving-cert: secret "maas-api-e2e-shared-b-w4-5adb63-serving-cert" not found Nov 03 09:34:49.389422 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:34:49.389414 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-maas-api-tls podName:74e77e1c-5e86-424c-bf70-6d34265486f9 nodeName:}" failed. No retries permitted until 2026-11-03 09:34:49.889392627 +0000 UTC m=+3322.814335035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-maas-api-tls") pod "maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" (UID: "74e77e1c-5e86-424c-bf70-6d34265486f9") : secret "maas-api-e2e-shared-b-w4-5adb63-serving-cert" not found Nov 03 09:34:49.389848 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.389828 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e77e1c-5e86-424c-bf70-6d34265486f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.391917 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.391895 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.448904 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.448826 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4bzz\" (UniqueName: \"kubernetes.io/projected/74e77e1c-5e86-424c-bf70-6d34265486f9-kube-api-access-q4bzz\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.895206 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.895164 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:49.898046 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:49.898009 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:50.094437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:50.094404 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:50.482759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:50.482727 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww"] Nov 03 09:34:50.483392 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:34:50.483361 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e77e1c_5e86_424c_bf70_6d34265486f9.slice/crio-985fc06daf96c3f337e9e2a3fc03605ca4a995f88ccf8c6c6821151c23da22cf WatchSource:0}: Error finding container 985fc06daf96c3f337e9e2a3fc03605ca4a995f88ccf8c6c6821151c23da22cf: Status 404 returned error can't find the container with id 985fc06daf96c3f337e9e2a3fc03605ca4a995f88ccf8c6c6821151c23da22cf Nov 03 09:34:51.123214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:51.123180 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" event={"ID":"74e77e1c-5e86-424c-bf70-6d34265486f9","Type":"ContainerStarted","Data":"985fc06daf96c3f337e9e2a3fc03605ca4a995f88ccf8c6c6821151c23da22cf"} Nov 03 09:34:52.132321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:52.132282 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" event={"ID":"74e77e1c-5e86-424c-bf70-6d34265486f9","Type":"ContainerStarted","Data":"dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d"} Nov 03 09:34:52.132758 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:52.132511 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:34:52.250959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:52.250902 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" podStartSLOduration=2.400008775 podStartE2EDuration="3.250883282s" totalImagesPullingTime="850.874507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:34:52.249622994 +0000 UTC m=+3325.174565229" watchObservedRunningTime="2026-11-03 09:34:52.250883282 +0000 UTC m=+3325.175825500" Nov 03 09:34:58.147615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:34:58.147582 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:35:09.247595 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.247547 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd"] Nov 03 09:35:09.248123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.247821 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" podUID="caaf72c6-18eb-4ec4-ac02-43f5d4095027" containerName="maas-api" containerID="cri-o://f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074" gracePeriod=30 Nov 03 09:35:09.546040 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.546013 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:35:09.657933 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.657899 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caaf72c6-18eb-4ec4-ac02-43f5d4095027-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caaf72c6-18eb-4ec4-ac02-43f5d4095027-openshift-service-ca\") pod \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " Nov 03 09:35:09.658142 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.657973 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caaf72c6-18eb-4ec4-ac02-43f5d4095027-kube-api-access-bkzrh\" (UniqueName: \"kubernetes.io/projected/caaf72c6-18eb-4ec4-ac02-43f5d4095027-kube-api-access-bkzrh\") pod \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " Nov 03 09:35:09.658142 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.658030 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-maas-api-tls\") pod \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " Nov 03 09:35:09.658142 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.658068 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-metrics-certs\" (UniqueName: \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-metrics-certs\") pod \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\" (UID: \"caaf72c6-18eb-4ec4-ac02-43f5d4095027\") " Nov 03 09:35:09.659054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.658750 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caaf72c6-18eb-4ec4-ac02-43f5d4095027-openshift-service-ca" pod "caaf72c6-18eb-4ec4-ac02-43f5d4095027" (UID: "caaf72c6-18eb-4ec4-ac02-43f5d4095027"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:35:09.661751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.661708 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-maas-api-tls" pod "caaf72c6-18eb-4ec4-ac02-43f5d4095027" (UID: "caaf72c6-18eb-4ec4-ac02-43f5d4095027"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:35:09.662115 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.662056 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-metrics-certs" pod "caaf72c6-18eb-4ec4-ac02-43f5d4095027" (UID: "caaf72c6-18eb-4ec4-ac02-43f5d4095027"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:35:09.670545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.670511 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caaf72c6-18eb-4ec4-ac02-43f5d4095027-kube-api-access-bkzrh" pod "caaf72c6-18eb-4ec4-ac02-43f5d4095027" (UID: "caaf72c6-18eb-4ec4-ac02-43f5d4095027"). InnerVolumeSpecName "kube-api-access-bkzrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:35:09.760159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.760072 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caaf72c6-18eb-4ec4-ac02-43f5d4095027-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:09.760159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.760103 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzrh\" (UniqueName: \"kubernetes.io/projected/caaf72c6-18eb-4ec4-ac02-43f5d4095027-kube-api-access-bkzrh\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:09.760159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.760113 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:09.760159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:09.760122 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caaf72c6-18eb-4ec4-ac02-43f5d4095027-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:10.275898 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:10.275858 5015 generic.go:356] "Generic (PLEG): container finished" podID="caaf72c6-18eb-4ec4-ac02-43f5d4095027" containerID="f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074" exitCode=0 Nov 03 09:35:10.276361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:10.275916 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" Nov 03 09:35:10.276361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:10.275953 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" event={"ID":"caaf72c6-18eb-4ec4-ac02-43f5d4095027","Type":"ContainerDied","Data":"f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074"} Nov 03 09:35:10.276361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:10.276027 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd" event={"ID":"caaf72c6-18eb-4ec4-ac02-43f5d4095027","Type":"ContainerDied","Data":"6d165d090bbf3ab1d29d5a5e5ecfc8c5f02019a38fc2dd161edc6252a5eb203d"} Nov 03 09:35:10.276361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:10.276059 5015 scope.go:122] "RemoveContainer" containerID="f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074" Nov 03 09:35:10.289551 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:10.289528 5015 scope.go:122] "RemoveContainer" containerID="f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074" Nov 03 09:35:10.290021 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:35:10.289992 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074\": container with ID starting with f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074 not found: ID does not exist" containerID="f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074" Nov 03 09:35:10.290102 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:10.290033 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074"} err="failed to get container status \"f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074\": rpc error: code = NotFound desc = could not find container \"f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074\": container with ID starting with f9a56573ddfcc9f93f6bda88a4ae6944006fd70045806ab11382b63aec317074 not found: ID does not exist" Nov 03 09:35:10.446774 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:10.446741 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd"] Nov 03 09:35:10.496779 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:10.496741 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f66c6c-6bd6cf77-h2pqd"] Nov 03 09:35:11.207988 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:11.207957 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caaf72c6-18eb-4ec4-ac02-43f5d4095027" path="/var/lib/kubelet/pods/caaf72c6-18eb-4ec4-ac02-43f5d4095027/volumes" Nov 03 09:35:16.903625 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:35:16.903580 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:35:34.348595 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.348554 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-558f97989d-qqg7n"] Nov 03 09:35:34.349569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.349542 5015 cpu_manager.go:414] "containerMap: removing container" podUID="caaf72c6-18eb-4ec4-ac02-43f5d4095027" containerName="maas-api" Nov 03 09:35:34.349569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.349568 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caaf72c6-18eb-4ec4-ac02-43f5d4095027" containerName="maas-api" Nov 03 09:35:34.349884 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.349855 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="caaf72c6-18eb-4ec4-ac02-43f5d4095027" containerName="maas-api" Nov 03 09:35:34.355424 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.355403 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.408216 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.408181 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-558f97989d-qqg7n"] Nov 03 09:35:34.454022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.453985 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b1c3ed9d233a\"" Nov 03 09:35:34.454022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.454000 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 09:35:34.454022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.454000 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 09:35:34.454356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.454111 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 09:35:34.454356 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.454173 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 09:35:34.459508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.459470 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-gkdr4\"" Nov 03 09:35:34.461082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.461051 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-routing-overlay-volume\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.461229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.461188 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vmk\" (UniqueName: \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-kube-api-access-r4vmk\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.461343 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.461325 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-plugins-config-volume\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.461410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.461384 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-provider-credentials\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.562421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.562381 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-plugins-config-volume\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.562670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.562450 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-provider-credentials\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.562670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.562522 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-routing-overlay-volume\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.562670 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.562620 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vmk\" (UniqueName: \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-kube-api-access-r4vmk\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.563302 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.563273 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-plugins-config-volume\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.563302 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.563293 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-routing-overlay-volume\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.565235 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.565210 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-provider-credentials\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.622426 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.622359 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vmk\" (UniqueName: \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-kube-api-access-r4vmk\") pod \"payload-processing-external-model-558f97989d-qqg7n\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:34.671565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:34.671533 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:35.053854 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:35:35.053805 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba87353_f551_4440_b835_9586a05ad1bf.slice/crio-87f76c8348a948fb2406b03c5ecf8461f0397db53d59d13a9b62fac6ba3fd193 WatchSource:0}: Error finding container 87f76c8348a948fb2406b03c5ecf8461f0397db53d59d13a9b62fac6ba3fd193: Status 404 returned error can't find the container with id 87f76c8348a948fb2406b03c5ecf8461f0397db53d59d13a9b62fac6ba3fd193 Nov 03 09:35:35.053986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:35.053898 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-558f97989d-qqg7n"] Nov 03 09:35:35.489647 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:35.489608 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" event={"ID":"8ba87353-f551-4440-b835-9586a05ad1bf","Type":"ContainerStarted","Data":"5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8"} Nov 03 09:35:35.490062 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:35.489675 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" event={"ID":"8ba87353-f551-4440-b835-9586a05ad1bf","Type":"ContainerStarted","Data":"87f76c8348a948fb2406b03c5ecf8461f0397db53d59d13a9b62fac6ba3fd193"} Nov 03 09:35:35.490062 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:35.489706 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:35.608680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:35.608634 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" podStartSLOduration=1.608619437 podStartE2EDuration="1.608619437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:35:35.60562693 +0000 UTC m=+3368.530569158" watchObservedRunningTime="2026-11-03 09:35:35.608619437 +0000 UTC m=+3368.533561653" Nov 03 09:35:46.502174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:35:46.502145 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:35:46.905839 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:35:46.905728 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:36:16.902647 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:36:16.902610 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:36:20.809582 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.809544 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb"] Nov 03 09:36:20.814173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.814156 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:20.868515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.868464 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pqxg4\"" Nov 03 09:36:20.922981 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.922952 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb"] Nov 03 09:36:20.937593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.937561 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:20.937741 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.937680 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:20.937786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.937757 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:20.937833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.937796 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:20.937939 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.937918 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:20.938009 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.937960 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:20.938069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.938043 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:20.938127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.938090 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab4552-a914-4ccc-a3e1-211981c0feae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:20.938246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:20.938218 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4775d\" (UniqueName: \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-kube-api-access-4775d\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.039713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.039676 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.039892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.039734 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.039892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.039787 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.039892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.039813 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.039892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.039858 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.039892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.039888 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.040196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.039941 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.040196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.039974 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab4552-a914-4ccc-a3e1-211981c0feae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.040196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.040042 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4775d\" (UniqueName: \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-kube-api-access-4775d\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.040375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.040254 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.040375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.040327 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.040375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.040362 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.040702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.040673 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.041046 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.041022 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab4552-a914-4ccc-a3e1-211981c0feae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.042560 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.042540 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.042653 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.042575 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.101285 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.101209 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.101862 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.101836 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4775d\" (UniqueName: \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-kube-api-access-4775d\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pgwkb\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.129838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.129811 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:21.568773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.568742 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb"] Nov 03 09:36:21.570540 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:36:21.570511 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ab4552_a914_4ccc_a3e1_211981c0feae.slice/crio-d9d53e84e4944197cf59add88e5715a2343e6c17a75bfdb9b0d0f93b24686e77 WatchSource:0}: Error finding container d9d53e84e4944197cf59add88e5715a2343e6c17a75bfdb9b0d0f93b24686e77: Status 404 returned error can't find the container with id d9d53e84e4944197cf59add88e5715a2343e6c17a75bfdb9b0d0f93b24686e77 Nov 03 09:36:21.573749 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.573713 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:36:21.573837 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.573783 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:36:21.573837 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.573813 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:36:21.590244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.590214 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw"] Nov 03 09:36:21.703893 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.703860 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b"] Nov 03 09:36:21.855136 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.855042 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" event={"ID":"94ab4552-a914-4ccc-a3e1-211981c0feae","Type":"ContainerStarted","Data":"01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203"} Nov 03 09:36:21.855136 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.855097 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" event={"ID":"94ab4552-a914-4ccc-a3e1-211981c0feae","Type":"ContainerStarted","Data":"d9d53e84e4944197cf59add88e5715a2343e6c17a75bfdb9b0d0f93b24686e77"} Nov 03 09:36:21.981388 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:21.981319 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" podStartSLOduration=1.9812995 podStartE2EDuration="1.9812995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:36:21.977738485 +0000 UTC m=+3414.902680730" watchObservedRunningTime="2026-11-03 09:36:21.9812995 +0000 UTC m=+3414.906241718" Nov 03 09:36:22.130276 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:22.130195 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:22.131403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:22.131381 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 09:36:22.131533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:22.131428 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 09:36:23.130383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:23.130348 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 09:36:23.130924 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:23.130407 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 09:36:24.130703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:24.130663 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 09:36:24.131149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:24.130733 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 09:36:25.130814 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:25.130773 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 09:36:25.131235 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:25.130852 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 09:36:26.130669 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:26.130625 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 09:36:26.130902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:26.130683 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 09:36:26.590991 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:26.590912 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" podUID="6d7a2217-86bb-4483-acb5-9125efc1f71d" containerName="payload-processing" containerID="cri-o://d6d5af0b79e5d3a9ed5b14622d3f118ebdfbfa2ced39fd994554d03bc14546d1" gracePeriod=25 Nov 03 09:36:26.704835 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:26.704784 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" podUID="bd6b8abd-22ac-4661-a14e-a7262397e9cb" containerName="payload-pre-processing" containerID="cri-o://bc1f60ef861a2e1f88ca55d9968c467d379899ab53bd1a0e385fbdf16c6bf7f7" gracePeriod=25 Nov 03 09:36:26.917727 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:26.917618 5015 generic.go:356] "Generic (PLEG): container finished" podID="6d7a2217-86bb-4483-acb5-9125efc1f71d" containerID="d6d5af0b79e5d3a9ed5b14622d3f118ebdfbfa2ced39fd994554d03bc14546d1" exitCode=0 Nov 03 09:36:26.917727 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:26.917693 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" event={"ID":"6d7a2217-86bb-4483-acb5-9125efc1f71d","Type":"ContainerDied","Data":"d6d5af0b79e5d3a9ed5b14622d3f118ebdfbfa2ced39fd994554d03bc14546d1"} Nov 03 09:36:26.917963 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:26.917768 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" event={"ID":"6d7a2217-86bb-4483-acb5-9125efc1f71d","Type":"ContainerDied","Data":"149a2db12ebd5bfbb2f4aeb0a01a92ab0643265d107a1987c0d960cd3c53d6c2"} Nov 03 09:36:26.917963 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:26.917793 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="149a2db12ebd5bfbb2f4aeb0a01a92ab0643265d107a1987c0d960cd3c53d6c2" Nov 03 09:36:26.919766 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:26.919736 5015 generic.go:356] "Generic (PLEG): container finished" podID="bd6b8abd-22ac-4661-a14e-a7262397e9cb" containerID="bc1f60ef861a2e1f88ca55d9968c467d379899ab53bd1a0e385fbdf16c6bf7f7" exitCode=0 Nov 03 09:36:26.919893 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:26.919807 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" event={"ID":"bd6b8abd-22ac-4661-a14e-a7262397e9cb","Type":"ContainerDied","Data":"bc1f60ef861a2e1f88ca55d9968c467d379899ab53bd1a0e385fbdf16c6bf7f7"} Nov 03 09:36:26.930154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:26.930129 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:36:27.071618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.071591 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:36:27.075337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.075308 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7a2217-86bb-4483-acb5-9125efc1f71d-kube-api-access-mzqmd\" (UniqueName: \"kubernetes.io/projected/6d7a2217-86bb-4483-acb5-9125efc1f71d-kube-api-access-mzqmd\") pod \"6d7a2217-86bb-4483-acb5-9125efc1f71d\" (UID: \"6d7a2217-86bb-4483-acb5-9125efc1f71d\") " Nov 03 09:36:27.075588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.075505 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d7a2217-86bb-4483-acb5-9125efc1f71d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7a2217-86bb-4483-acb5-9125efc1f71d-plugins-config-volume\") pod \"6d7a2217-86bb-4483-acb5-9125efc1f71d\" (UID: \"6d7a2217-86bb-4483-acb5-9125efc1f71d\") " Nov 03 09:36:27.075976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.075944 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7a2217-86bb-4483-acb5-9125efc1f71d-plugins-config-volume" pod "6d7a2217-86bb-4483-acb5-9125efc1f71d" (UID: "6d7a2217-86bb-4483-acb5-9125efc1f71d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:36:27.076195 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.076155 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7a2217-86bb-4483-acb5-9125efc1f71d-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:27.077817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.077793 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7a2217-86bb-4483-acb5-9125efc1f71d-kube-api-access-mzqmd" pod "6d7a2217-86bb-4483-acb5-9125efc1f71d" (UID: "6d7a2217-86bb-4483-acb5-9125efc1f71d"). InnerVolumeSpecName "kube-api-access-mzqmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:36:27.130418 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.130385 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 09:36:27.130663 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.130439 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 09:36:27.177049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.176953 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6b8abd-22ac-4661-a14e-a7262397e9cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6b8abd-22ac-4661-a14e-a7262397e9cb-plugins-config-volume\") pod \"bd6b8abd-22ac-4661-a14e-a7262397e9cb\" (UID: \"bd6b8abd-22ac-4661-a14e-a7262397e9cb\") " Nov 03 09:36:27.177049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.176996 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6b8abd-22ac-4661-a14e-a7262397e9cb-kube-api-access-mpwsk\" (UniqueName: \"kubernetes.io/projected/bd6b8abd-22ac-4661-a14e-a7262397e9cb-kube-api-access-mpwsk\") pod \"bd6b8abd-22ac-4661-a14e-a7262397e9cb\" (UID: \"bd6b8abd-22ac-4661-a14e-a7262397e9cb\") " Nov 03 09:36:27.177485 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.177314 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6b8abd-22ac-4661-a14e-a7262397e9cb-plugins-config-volume" pod "bd6b8abd-22ac-4661-a14e-a7262397e9cb" (UID: "bd6b8abd-22ac-4661-a14e-a7262397e9cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:36:27.177485 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.177387 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzqmd\" (UniqueName: \"kubernetes.io/projected/6d7a2217-86bb-4483-acb5-9125efc1f71d-kube-api-access-mzqmd\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:27.179301 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.179277 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6b8abd-22ac-4661-a14e-a7262397e9cb-kube-api-access-mpwsk" pod "bd6b8abd-22ac-4661-a14e-a7262397e9cb" (UID: "bd6b8abd-22ac-4661-a14e-a7262397e9cb"). InnerVolumeSpecName "kube-api-access-mpwsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:36:27.278160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.278128 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6b8abd-22ac-4661-a14e-a7262397e9cb-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:27.278309 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.278156 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpwsk\" (UniqueName: \"kubernetes.io/projected/bd6b8abd-22ac-4661-a14e-a7262397e9cb-kube-api-access-mpwsk\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:27.928671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.928585 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" Nov 03 09:36:27.928671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.928628 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b" event={"ID":"bd6b8abd-22ac-4661-a14e-a7262397e9cb","Type":"ContainerDied","Data":"acb5e646ccec59351ac0643effc3a5f1178e007e5e79c80379616dd736941d69"} Nov 03 09:36:27.928902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.928700 5015 scope.go:122] "RemoveContainer" containerID="bc1f60ef861a2e1f88ca55d9968c467d379899ab53bd1a0e385fbdf16c6bf7f7" Nov 03 09:36:27.928902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:27.928759 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw" Nov 03 09:36:28.130631 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:28.130597 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 09:36:28.130803 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:28.130647 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 09:36:28.156986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:28.156957 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b"] Nov 03 09:36:28.160986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:28.160960 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03a6e5-5b47b9c6d7-x692b"] Nov 03 09:36:28.273059 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:28.273023 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw"] Nov 03 09:36:28.284140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:28.284107 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03a6e5-578b89dd5-8hnmw"] Nov 03 09:36:28.683537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:28.683417 5015 scope.go:122] "RemoveContainer" containerID="d6d5af0b79e5d3a9ed5b14622d3f118ebdfbfa2ced39fd994554d03bc14546d1" Nov 03 09:36:29.131158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:29.131120 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 09:36:29.131387 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:29.131190 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 09:36:29.207740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:29.207706 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d7a2217-86bb-4483-acb5-9125efc1f71d" path="/var/lib/kubelet/pods/6d7a2217-86bb-4483-acb5-9125efc1f71d/volumes" Nov 03 09:36:29.208097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:29.208079 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6b8abd-22ac-4661-a14e-a7262397e9cb" path="/var/lib/kubelet/pods/bd6b8abd-22ac-4661-a14e-a7262397e9cb/volumes" Nov 03 09:36:30.384839 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:30.384798 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:30.385791 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:30.385743 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:30.386782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:30.386761 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:36:37.966904 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:37.966868 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd"] Nov 03 09:36:37.967594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:37.967579 5015 cpu_manager.go:414] "containerMap: removing container" podUID="bd6b8abd-22ac-4661-a14e-a7262397e9cb" containerName="payload-pre-processing" Nov 03 09:36:37.967710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:37.967596 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6b8abd-22ac-4661-a14e-a7262397e9cb" containerName="payload-pre-processing" Nov 03 09:36:37.967710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:37.967630 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6d7a2217-86bb-4483-acb5-9125efc1f71d" containerName="payload-processing" Nov 03 09:36:37.967710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:37.967635 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7a2217-86bb-4483-acb5-9125efc1f71d" containerName="payload-processing" Nov 03 09:36:37.967809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:37.967739 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7a2217-86bb-4483-acb5-9125efc1f71d" containerName="payload-processing" Nov 03 09:36:37.967809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:37.967755 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6b8abd-22ac-4661-a14e-a7262397e9cb" containerName="payload-pre-processing" Nov 03 09:36:37.972890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:37.972872 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.016523 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.016470 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-plugins-config-volume\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.016771 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.016745 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-routing-overlay-volume\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.016852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.016797 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-provider-credentials\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.016979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.016946 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48gzd\" (UniqueName: \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-kube-api-access-48gzd\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.028185 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.028146 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd"] Nov 03 09:36:38.073841 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.073812 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-546e9f5ec78c\"" Nov 03 09:36:38.117774 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.117734 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-routing-overlay-volume\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.117961 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.117782 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-provider-credentials\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.117961 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.117835 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48gzd\" (UniqueName: \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-kube-api-access-48gzd\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.117961 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.117871 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-plugins-config-volume\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.118552 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.118520 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-routing-overlay-volume\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.118814 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.118793 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-plugins-config-volume\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.120641 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.120621 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-provider-credentials\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.177270 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.177230 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48gzd\" (UniqueName: \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-kube-api-access-48gzd\") pod \"payload-processing-external-model-6cdb8bd7fb-4bxdd\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.287001 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.286969 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:38.673826 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:36:38.673794 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aba9632_f731_4763_b9c0_dd2a42874d17.slice/crio-eaab46d10752fd1e45b90fe8730561994214c1e7105be898682756898d543201 WatchSource:0}: Error finding container eaab46d10752fd1e45b90fe8730561994214c1e7105be898682756898d543201: Status 404 returned error can't find the container with id eaab46d10752fd1e45b90fe8730561994214c1e7105be898682756898d543201 Nov 03 09:36:38.674266 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:38.674239 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd"] Nov 03 09:36:39.016914 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:39.016876 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" event={"ID":"0aba9632-f731-4763-b9c0-dd2a42874d17","Type":"ContainerStarted","Data":"d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7"} Nov 03 09:36:39.017420 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:39.016935 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" event={"ID":"0aba9632-f731-4763-b9c0-dd2a42874d17","Type":"ContainerStarted","Data":"eaab46d10752fd1e45b90fe8730561994214c1e7105be898682756898d543201"} Nov 03 09:36:39.017420 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:39.016978 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:39.135425 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:39.135377 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" podStartSLOduration=2.135363067 podStartE2EDuration="2.135363067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:36:39.134722464 +0000 UTC m=+3432.059664706" watchObservedRunningTime="2026-11-03 09:36:39.135363067 +0000 UTC m=+3432.060305283" Nov 03 09:36:46.918091 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:36:46.918054 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:36:50.028333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:50.028295 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:36:50.205783 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:50.205746 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-558f97989d-qqg7n"] Nov 03 09:36:55.206698 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.206656 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" podUID="8ba87353-f551-4440-b835-9586a05ad1bf" containerName="payload-processing" containerID="cri-o://5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8" gracePeriod=25 Nov 03 09:36:55.491601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.491567 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:36:55.553787 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.553749 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-routing-overlay-volume\") pod \"8ba87353-f551-4440-b835-9586a05ad1bf\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " Nov 03 09:36:55.553982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.553812 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-kube-api-access-r4vmk\" (UniqueName: \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-kube-api-access-r4vmk\") pod \"8ba87353-f551-4440-b835-9586a05ad1bf\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " Nov 03 09:36:55.553982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.553942 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-provider-credentials\") pod \"8ba87353-f551-4440-b835-9586a05ad1bf\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " Nov 03 09:36:55.554158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.554125 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-routing-overlay-volume" pod "8ba87353-f551-4440-b835-9586a05ad1bf" (UID: "8ba87353-f551-4440-b835-9586a05ad1bf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:36:55.554319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.554195 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-plugins-config-volume\") pod \"8ba87353-f551-4440-b835-9586a05ad1bf\" (UID: \"8ba87353-f551-4440-b835-9586a05ad1bf\") " Nov 03 09:36:55.554806 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.554657 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-plugins-config-volume" pod "8ba87353-f551-4440-b835-9586a05ad1bf" (UID: "8ba87353-f551-4440-b835-9586a05ad1bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:36:55.556509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.555084 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:55.556509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.555107 5015 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ba87353-f551-4440-b835-9586a05ad1bf-routing-overlay-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:55.556685 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.556574 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-provider-credentials" pod "8ba87353-f551-4440-b835-9586a05ad1bf" (UID: "8ba87353-f551-4440-b835-9586a05ad1bf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:36:55.556755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.556729 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-kube-api-access-r4vmk" pod "8ba87353-f551-4440-b835-9586a05ad1bf" (UID: "8ba87353-f551-4440-b835-9586a05ad1bf"). InnerVolumeSpecName "kube-api-access-r4vmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:36:55.656113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.656083 5015 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-provider-credentials\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:55.656113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:55.656109 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4vmk\" (UniqueName: \"kubernetes.io/projected/8ba87353-f551-4440-b835-9586a05ad1bf-kube-api-access-r4vmk\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:56.165897 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:56.165858 5015 generic.go:356] "Generic (PLEG): container finished" podID="8ba87353-f551-4440-b835-9586a05ad1bf" containerID="5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8" exitCode=0 Nov 03 09:36:56.166113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:56.165921 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" Nov 03 09:36:56.166113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:56.165954 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" event={"ID":"8ba87353-f551-4440-b835-9586a05ad1bf","Type":"ContainerDied","Data":"5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8"} Nov 03 09:36:56.166113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:56.166032 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-558f97989d-qqg7n" event={"ID":"8ba87353-f551-4440-b835-9586a05ad1bf","Type":"ContainerDied","Data":"87f76c8348a948fb2406b03c5ecf8461f0397db53d59d13a9b62fac6ba3fd193"} Nov 03 09:36:56.166113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:56.166070 5015 scope.go:122] "RemoveContainer" containerID="5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8" Nov 03 09:36:56.178601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:56.178582 5015 scope.go:122] "RemoveContainer" containerID="5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8" Nov 03 09:36:56.178884 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:36:56.178865 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8\": container with ID starting with 5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8 not found: ID does not exist" containerID="5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8" Nov 03 09:36:56.178925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:56.178894 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8"} err="failed to get container status \"5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8\": rpc error: code = NotFound desc = could not find container \"5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8\": container with ID starting with 5f1f7be881c6087ea2233a2b4d2a290e5c0ccb671e745e27de9541c40bc989a8 not found: ID does not exist" Nov 03 09:36:56.345718 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:56.344912 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-558f97989d-qqg7n"] Nov 03 09:36:56.398500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:56.396207 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-558f97989d-qqg7n"] Nov 03 09:36:57.206570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:36:57.206531 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba87353-f551-4440-b835-9586a05ad1bf" path="/var/lib/kubelet/pods/8ba87353-f551-4440-b835-9586a05ad1bf/volumes" Nov 03 09:37:03.268608 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:03.268571 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd"] Nov 03 09:37:06.157158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.157068 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f"] Nov 03 09:37:06.157982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.157959 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8ba87353-f551-4440-b835-9586a05ad1bf" containerName="payload-processing" Nov 03 09:37:06.158095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.157983 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba87353-f551-4440-b835-9586a05ad1bf" containerName="payload-processing" Nov 03 09:37:06.158201 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.158186 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba87353-f551-4440-b835-9586a05ad1bf" containerName="payload-processing" Nov 03 09:37:06.162055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.162035 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.195653 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.195622 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.195802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.195722 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.195802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.195756 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.195905 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.195886 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.195958 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.195943 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.196053 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.196035 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb2f094-67e8-438b-80b8-c1ef4521b355-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.196090 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.196064 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbp2\" (UniqueName: \"kubernetes.io/projected/fdb2f094-67e8-438b-80b8-c1ef4521b355-kube-api-access-nmbp2\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.196131 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.196087 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.196166 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.196158 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.297366 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.297329 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.297581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.297408 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.297581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.297457 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.297581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.297524 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb2f094-67e8-438b-80b8-c1ef4521b355-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.297581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.297552 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbp2\" (UniqueName: \"kubernetes.io/projected/fdb2f094-67e8-438b-80b8-c1ef4521b355-kube-api-access-nmbp2\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.297581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.297578 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.297926 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.297624 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.297926 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.297697 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.297926 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.297769 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.297926 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.297804 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.297926 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.297855 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.298172 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.298155 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.298245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.298223 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.298286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.298248 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb2f094-67e8-438b-80b8-c1ef4521b355-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.300332 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.300311 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.300436 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.300418 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.357499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.357439 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb2f094-67e8-438b-80b8-c1ef4521b355-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.357712 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.357573 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbp2\" (UniqueName: \"kubernetes.io/projected/fdb2f094-67e8-438b-80b8-c1ef4521b355-kube-api-access-nmbp2\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lkg9f\" (UID: \"fdb2f094-67e8-438b-80b8-c1ef4521b355\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.366387 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.366360 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f"] Nov 03 09:37:06.476651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.476622 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:06.914955 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.914921 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f"] Nov 03 09:37:06.915632 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:37:06.915609 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb2f094_67e8_438b_80b8_c1ef4521b355.slice/crio-f7b2c49da27d879fdb4caf6ea98124ea0edfec2a5c4ac312a5ddfa0e1cba0fb1 WatchSource:0}: Error finding container f7b2c49da27d879fdb4caf6ea98124ea0edfec2a5c4ac312a5ddfa0e1cba0fb1: Status 404 returned error can't find the container with id f7b2c49da27d879fdb4caf6ea98124ea0edfec2a5c4ac312a5ddfa0e1cba0fb1 Nov 03 09:37:06.918178 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.918144 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:37:06.918269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.918213 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:37:06.918269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:06.918242 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:37:07.253087 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:07.253040 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" event={"ID":"fdb2f094-67e8-438b-80b8-c1ef4521b355","Type":"ContainerStarted","Data":"345390a61f7fc4bec73bbd747cda06a4a69718d7c9862b3ec9ab6310e6fde423"} Nov 03 09:37:07.253623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:07.253107 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" event={"ID":"fdb2f094-67e8-438b-80b8-c1ef4521b355","Type":"ContainerStarted","Data":"f7b2c49da27d879fdb4caf6ea98124ea0edfec2a5c4ac312a5ddfa0e1cba0fb1"} Nov 03 09:37:07.374529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:07.374447 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" podStartSLOduration=1.374429745 podStartE2EDuration="1.374429745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:07.372175018 +0000 UTC m=+3460.297117246" watchObservedRunningTime="2026-11-03 09:37:07.374429745 +0000 UTC m=+3460.299371964" Nov 03 09:37:07.477651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:07.477609 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:07.479002 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:07.478974 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 09:37:07.479146 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:07.479053 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" podUID="fdb2f094-67e8-438b-80b8-c1ef4521b355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 09:37:08.269877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.269841 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" podUID="0aba9632-f731-4763-b9c0-dd2a42874d17" containerName="payload-processing" containerID="cri-o://d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7" gracePeriod=25 Nov 03 09:37:08.481806 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.481772 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 09:37:08.483301 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.483275 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" podUID="fdb2f094-67e8-438b-80b8-c1ef4521b355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 09:37:08.586079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.586053 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:37:08.629782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.629754 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-provider-credentials\") pod \"0aba9632-f731-4763-b9c0-dd2a42874d17\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " Nov 03 09:37:08.629966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.629796 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-plugins-config-volume\") pod \"0aba9632-f731-4763-b9c0-dd2a42874d17\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " Nov 03 09:37:08.629966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.629898 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-kube-api-access-48gzd\" (UniqueName: \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-kube-api-access-48gzd\") pod \"0aba9632-f731-4763-b9c0-dd2a42874d17\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " Nov 03 09:37:08.630093 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.629975 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-routing-overlay-volume\") pod \"0aba9632-f731-4763-b9c0-dd2a42874d17\" (UID: \"0aba9632-f731-4763-b9c0-dd2a42874d17\") " Nov 03 09:37:08.630330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.630303 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-routing-overlay-volume" pod "0aba9632-f731-4763-b9c0-dd2a42874d17" (UID: "0aba9632-f731-4763-b9c0-dd2a42874d17"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:37:08.630330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.630299 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-plugins-config-volume" pod "0aba9632-f731-4763-b9c0-dd2a42874d17" (UID: "0aba9632-f731-4763-b9c0-dd2a42874d17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:37:08.630531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.630447 5015 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-routing-overlay-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:08.630531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.630468 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aba9632-f731-4763-b9c0-dd2a42874d17-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:08.632251 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.632222 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-provider-credentials" pod "0aba9632-f731-4763-b9c0-dd2a42874d17" (UID: "0aba9632-f731-4763-b9c0-dd2a42874d17"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:37:08.632352 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.632263 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-kube-api-access-48gzd" pod "0aba9632-f731-4763-b9c0-dd2a42874d17" (UID: "0aba9632-f731-4763-b9c0-dd2a42874d17"). InnerVolumeSpecName "kube-api-access-48gzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:37:08.732041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.732008 5015 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-provider-credentials\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:08.732041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:08.732044 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48gzd\" (UniqueName: \"kubernetes.io/projected/0aba9632-f731-4763-b9c0-dd2a42874d17-kube-api-access-48gzd\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:09.275364 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:09.275328 5015 generic.go:356] "Generic (PLEG): container finished" podID="0aba9632-f731-4763-b9c0-dd2a42874d17" containerID="d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7" exitCode=0 Nov 03 09:37:09.275852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:09.275372 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" Nov 03 09:37:09.275852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:09.275417 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" event={"ID":"0aba9632-f731-4763-b9c0-dd2a42874d17","Type":"ContainerDied","Data":"d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7"} Nov 03 09:37:09.275852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:09.275501 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd" event={"ID":"0aba9632-f731-4763-b9c0-dd2a42874d17","Type":"ContainerDied","Data":"eaab46d10752fd1e45b90fe8730561994214c1e7105be898682756898d543201"} Nov 03 09:37:09.275852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:09.275536 5015 scope.go:122] "RemoveContainer" containerID="d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7" Nov 03 09:37:09.287190 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:09.287172 5015 scope.go:122] "RemoveContainer" containerID="d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7" Nov 03 09:37:09.287500 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:37:09.287457 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7\": container with ID starting with d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7 not found: ID does not exist" containerID="d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7" Nov 03 09:37:09.287613 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:09.287508 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7"} err="failed to get container status \"d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7\": rpc error: code = NotFound desc = could not find container \"d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7\": container with ID starting with d04de54c33b6224d66170b41733dde00ff63a7357beffbf3832d58b83123aac7 not found: ID does not exist" Nov 03 09:37:09.469168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:09.468283 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd"] Nov 03 09:37:09.477802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:09.477771 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 09:37:09.477940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:09.477831 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" podUID="fdb2f094-67e8-438b-80b8-c1ef4521b355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 09:37:09.508923 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:09.508895 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdb8bd7fb-4bxdd"] Nov 03 09:37:10.477015 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:10.476979 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 09:37:10.477466 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:10.477048 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" podUID="fdb2f094-67e8-438b-80b8-c1ef4521b355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 09:37:11.204905 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:11.204871 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aba9632-f731-4763-b9c0-dd2a42874d17" path="/var/lib/kubelet/pods/0aba9632-f731-4763-b9c0-dd2a42874d17/volumes" Nov 03 09:37:11.477024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:11.476991 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 09:37:11.477024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:11.477059 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" podUID="fdb2f094-67e8-438b-80b8-c1ef4521b355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 09:37:12.478449 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:12.478414 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 09:37:12.478953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:12.478470 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" podUID="fdb2f094-67e8-438b-80b8-c1ef4521b355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 09:37:13.483497 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:13.477638 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 09:37:13.483497 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:13.477715 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" podUID="fdb2f094-67e8-438b-80b8-c1ef4521b355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 09:37:14.478121 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:14.478078 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 09:37:14.478312 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:14.478141 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" podUID="fdb2f094-67e8-438b-80b8-c1ef4521b355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 09:37:15.316268 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.316232 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr"] Nov 03 09:37:15.317390 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.317362 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0aba9632-f731-4763-b9c0-dd2a42874d17" containerName="payload-processing" Nov 03 09:37:15.317553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.317389 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aba9632-f731-4763-b9c0-dd2a42874d17" containerName="payload-processing" Nov 03 09:37:15.317616 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.317601 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aba9632-f731-4763-b9c0-dd2a42874d17" containerName="payload-processing" Nov 03 09:37:15.323398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.323376 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.378262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.378230 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr"] Nov 03 09:37:15.420183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.420145 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 09:37:15.420354 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.420145 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 09:37:15.421395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.421368 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 09:37:15.421395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.421374 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-05110bbd79d3\"" Nov 03 09:37:15.421730 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.421703 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-gkdr4\"" Nov 03 09:37:15.427456 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.427432 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-546e9f5ec78c\"" Nov 03 09:37:15.427584 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.427494 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 09:37:15.455317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.455277 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-routing-overlay-volume\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.455464 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.455443 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6zj\" (UniqueName: \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-kube-api-access-tg6zj\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.455575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.455521 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-provider-credentials\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.455657 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.455633 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-plugins-config-volume\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.557173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.557136 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-routing-overlay-volume\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.557359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.557271 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6zj\" (UniqueName: \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-kube-api-access-tg6zj\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.557359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.557330 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-provider-credentials\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.557457 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.557369 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-plugins-config-volume\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.557977 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.557944 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-routing-overlay-volume\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.558412 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.558382 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-plugins-config-volume\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.560296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.560268 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-provider-credentials\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.616743 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.616662 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6zj\" (UniqueName: \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-kube-api-access-tg6zj\") pod \"payload-processing-external-model-f5bd745bd-ssdtr\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.642259 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.642224 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:15.755249 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.754604 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:15.755794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.755732 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:15.757024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:15.757000 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" Nov 03 09:37:16.054961 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:16.054930 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr"] Nov 03 09:37:16.056218 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:37:16.056188 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6611aae8_9bf5_44e7_a477_060ac9a7fb86.slice/crio-852b11b05e9da95e1dbbc8f6f2b9d7135674584b6f085a02dd67fabff224513a WatchSource:0}: Error finding container 852b11b05e9da95e1dbbc8f6f2b9d7135674584b6f085a02dd67fabff224513a: Status 404 returned error can't find the container with id 852b11b05e9da95e1dbbc8f6f2b9d7135674584b6f085a02dd67fabff224513a Nov 03 09:37:16.330149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:16.330054 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" event={"ID":"6611aae8-9bf5-44e7-a477-060ac9a7fb86","Type":"ContainerStarted","Data":"a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1"} Nov 03 09:37:16.330149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:16.330121 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" event={"ID":"6611aae8-9bf5-44e7-a477-060ac9a7fb86","Type":"ContainerStarted","Data":"852b11b05e9da95e1dbbc8f6f2b9d7135674584b6f085a02dd67fabff224513a"} Nov 03 09:37:16.468563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:16.468508 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podStartSLOduration=1.468492489 podStartE2EDuration="1.468492489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:16.464791712 +0000 UTC m=+3469.389733954" watchObservedRunningTime="2026-11-03 09:37:16.468492489 +0000 UTC m=+3469.393434703" Nov 03 09:37:16.904747 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:37:16.904705 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:37:17.338960 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:17.338923 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:37:21.281128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.281095 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7"] Nov 03 09:37:21.286618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.286597 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.434604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.434564 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.434840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.434717 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.434840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.434750 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.434840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.434811 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.435005 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.434861 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.435005 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.434880 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.435005 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.434990 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.435165 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.435018 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.435165 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.435059 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxgtj\" (UniqueName: \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-kube-api-access-hxgtj\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.489977 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.489944 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7"] Nov 03 09:37:21.536723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.536646 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.536723 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.536718 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.536929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.536735 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.536929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.536760 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.536929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.536790 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.536929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.536835 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.536929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.536885 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.536929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.536909 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.537264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.536958 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxgtj\" (UniqueName: \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-kube-api-access-hxgtj\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.537364 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.537332 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.537435 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.537375 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.537574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.537545 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.537710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.537691 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.538000 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.537975 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.539319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.539292 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.539415 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.539390 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.596085 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.596055 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.596206 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.596177 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxgtj\" (UniqueName: \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-kube-api-access-hxgtj\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d6bq7\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:21.601562 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:21.601538 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:22.036952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:22.036926 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7"] Nov 03 09:37:22.037529 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:37:22.037465 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84b7f8b_0f44_4c2c_86e9_9198d73399f3.slice/crio-c7b3bce9df64f202a1233e789d6db53e6da8d529992db600be908635565bab59 WatchSource:0}: Error finding container c7b3bce9df64f202a1233e789d6db53e6da8d529992db600be908635565bab59: Status 404 returned error can't find the container with id c7b3bce9df64f202a1233e789d6db53e6da8d529992db600be908635565bab59 Nov 03 09:37:22.040260 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:22.040228 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:37:22.040340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:22.040294 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:37:22.040340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:22.040323 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:37:22.382020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:22.381941 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" event={"ID":"e84b7f8b-0f44-4c2c-86e9-9198d73399f3","Type":"ContainerStarted","Data":"38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177"} Nov 03 09:37:22.382020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:22.382001 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" event={"ID":"e84b7f8b-0f44-4c2c-86e9-9198d73399f3","Type":"ContainerStarted","Data":"c7b3bce9df64f202a1233e789d6db53e6da8d529992db600be908635565bab59"} Nov 03 09:37:22.504625 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:22.504569 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" podStartSLOduration=1.504552653 podStartE2EDuration="1.504552653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:22.5013036 +0000 UTC m=+3475.426245846" watchObservedRunningTime="2026-11-03 09:37:22.504552653 +0000 UTC m=+3475.429494871" Nov 03 09:37:22.602129 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:22.602098 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:23.603797 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:23.603758 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 09:37:23.604193 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:23.603813 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 09:37:24.602049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:24.602012 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 09:37:24.602214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:24.602075 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 09:37:25.602462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:25.602419 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 09:37:25.602903 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:25.602520 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 09:37:26.602416 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:26.602375 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 09:37:26.602761 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:26.602445 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 09:37:27.601917 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:27.601875 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 09:37:27.602102 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:27.601934 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 09:37:28.351072 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:28.351023 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:37:28.602054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:28.601968 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 09:37:28.602054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:28.602039 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 09:37:29.602241 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:29.602205 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 09:37:29.602681 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:29.602259 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 09:37:31.020376 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:31.020331 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:31.020969 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:31.020779 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:31.021752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:31.021731 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:37:38.348870 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:38.348825 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:37:41.861713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:41.861681 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww"] Nov 03 09:37:41.862133 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:41.861893 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" podUID="74e77e1c-5e86-424c-bf70-6d34265486f9" containerName="maas-api" containerID="cri-o://dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d" gracePeriod=30 Nov 03 09:37:41.914930 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:41.914894 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr"] Nov 03 09:37:42.137168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.137145 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:37:42.259756 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.259724 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-maas-api-tls\") pod \"74e77e1c-5e86-424c-bf70-6d34265486f9\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " Nov 03 09:37:42.259964 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.259892 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-metrics-certs\") pod \"74e77e1c-5e86-424c-bf70-6d34265486f9\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " Nov 03 09:37:42.260054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.259963 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e77e1c-5e86-424c-bf70-6d34265486f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e77e1c-5e86-424c-bf70-6d34265486f9-openshift-service-ca\") pod \"74e77e1c-5e86-424c-bf70-6d34265486f9\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " Nov 03 09:37:42.260054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.260034 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e77e1c-5e86-424c-bf70-6d34265486f9-kube-api-access-q4bzz\" (UniqueName: \"kubernetes.io/projected/74e77e1c-5e86-424c-bf70-6d34265486f9-kube-api-access-q4bzz\") pod \"74e77e1c-5e86-424c-bf70-6d34265486f9\" (UID: \"74e77e1c-5e86-424c-bf70-6d34265486f9\") " Nov 03 09:37:42.261100 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.261065 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e77e1c-5e86-424c-bf70-6d34265486f9-openshift-service-ca" pod "74e77e1c-5e86-424c-bf70-6d34265486f9" (UID: "74e77e1c-5e86-424c-bf70-6d34265486f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:37:42.262209 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.262185 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-maas-api-tls" pod "74e77e1c-5e86-424c-bf70-6d34265486f9" (UID: "74e77e1c-5e86-424c-bf70-6d34265486f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:37:42.262409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.262391 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e77e1c-5e86-424c-bf70-6d34265486f9-kube-api-access-q4bzz" pod "74e77e1c-5e86-424c-bf70-6d34265486f9" (UID: "74e77e1c-5e86-424c-bf70-6d34265486f9"). InnerVolumeSpecName "kube-api-access-q4bzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:37:42.262505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.262424 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-metrics-certs" pod "74e77e1c-5e86-424c-bf70-6d34265486f9" (UID: "74e77e1c-5e86-424c-bf70-6d34265486f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:37:42.363192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.362778 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.363192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.363112 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e77e1c-5e86-424c-bf70-6d34265486f9-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.363192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.363151 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4bzz\" (UniqueName: \"kubernetes.io/projected/74e77e1c-5e86-424c-bf70-6d34265486f9-kube-api-access-q4bzz\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.363192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.363168 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e77e1c-5e86-424c-bf70-6d34265486f9-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.581252 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.581216 5015 generic.go:356] "Generic (PLEG): container finished" podID="74e77e1c-5e86-424c-bf70-6d34265486f9" containerID="dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d" exitCode=0 Nov 03 09:37:42.581451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.581262 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" Nov 03 09:37:42.581451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.581303 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" event={"ID":"74e77e1c-5e86-424c-bf70-6d34265486f9","Type":"ContainerDied","Data":"dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d"} Nov 03 09:37:42.581451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.581360 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww" event={"ID":"74e77e1c-5e86-424c-bf70-6d34265486f9","Type":"ContainerDied","Data":"985fc06daf96c3f337e9e2a3fc03605ca4a995f88ccf8c6c6821151c23da22cf"} Nov 03 09:37:42.581451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.581382 5015 scope.go:122] "RemoveContainer" containerID="dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d" Nov 03 09:37:42.592930 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.592911 5015 scope.go:122] "RemoveContainer" containerID="dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d" Nov 03 09:37:42.593187 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:37:42.593167 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d\": container with ID starting with dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d not found: ID does not exist" containerID="dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d" Nov 03 09:37:42.593235 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.593196 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d"} err="failed to get container status \"dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d\": rpc error: code = NotFound desc = could not find container \"dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d\": container with ID starting with dfb747d1d1c7886da741b1e0532892eff0c0578cbcd8ce4134f78a4b0847fa6d not found: ID does not exist" Nov 03 09:37:42.752029 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.751998 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww"] Nov 03 09:37:42.757527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:42.757498 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5adb63-c6d4b664f-fw9ww"] Nov 03 09:37:43.205511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:43.205453 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e77e1c-5e86-424c-bf70-6d34265486f9" path="/var/lib/kubelet/pods/74e77e1c-5e86-424c-bf70-6d34265486f9/volumes" Nov 03 09:37:46.903065 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:37:46.903017 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:37:46.915702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:46.915665 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" podUID="00af5a6c-af44-4a21-b197-c25b51ece5df" containerName="payload-processing" containerID="cri-o://0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb" gracePeriod=25 Nov 03 09:37:47.202633 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.202603 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:37:47.327384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.327354 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00af5a6c-af44-4a21-b197-c25b51ece5df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00af5a6c-af44-4a21-b197-c25b51ece5df-plugins-config-volume\") pod \"00af5a6c-af44-4a21-b197-c25b51ece5df\" (UID: \"00af5a6c-af44-4a21-b197-c25b51ece5df\") " Nov 03 09:37:47.327596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.327532 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00af5a6c-af44-4a21-b197-c25b51ece5df-kube-api-access-gc2x7\" (UniqueName: \"kubernetes.io/projected/00af5a6c-af44-4a21-b197-c25b51ece5df-kube-api-access-gc2x7\") pod \"00af5a6c-af44-4a21-b197-c25b51ece5df\" (UID: \"00af5a6c-af44-4a21-b197-c25b51ece5df\") " Nov 03 09:37:47.327814 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.327789 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00af5a6c-af44-4a21-b197-c25b51ece5df-plugins-config-volume" pod "00af5a6c-af44-4a21-b197-c25b51ece5df" (UID: "00af5a6c-af44-4a21-b197-c25b51ece5df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:37:47.328420 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.328382 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00af5a6c-af44-4a21-b197-c25b51ece5df-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:47.330640 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.330614 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00af5a6c-af44-4a21-b197-c25b51ece5df-kube-api-access-gc2x7" pod "00af5a6c-af44-4a21-b197-c25b51ece5df" (UID: "00af5a6c-af44-4a21-b197-c25b51ece5df"). InnerVolumeSpecName "kube-api-access-gc2x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:37:47.429804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.429730 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc2x7\" (UniqueName: \"kubernetes.io/projected/00af5a6c-af44-4a21-b197-c25b51ece5df-kube-api-access-gc2x7\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:47.633085 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.633048 5015 generic.go:356] "Generic (PLEG): container finished" podID="00af5a6c-af44-4a21-b197-c25b51ece5df" containerID="0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb" exitCode=0 Nov 03 09:37:47.633289 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.633110 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" Nov 03 09:37:47.633289 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.633136 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" event={"ID":"00af5a6c-af44-4a21-b197-c25b51ece5df","Type":"ContainerDied","Data":"0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb"} Nov 03 09:37:47.633289 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.633215 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr" event={"ID":"00af5a6c-af44-4a21-b197-c25b51ece5df","Type":"ContainerDied","Data":"6ea8576514b581855ccf702ae41c115b4320245df5d24bfdc0335396e90c653b"} Nov 03 09:37:47.633289 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.633237 5015 scope.go:122] "RemoveContainer" containerID="0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb" Nov 03 09:37:47.646177 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.646156 5015 scope.go:122] "RemoveContainer" containerID="0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb" Nov 03 09:37:47.646457 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:37:47.646430 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb\": container with ID starting with 0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb not found: ID does not exist" containerID="0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb" Nov 03 09:37:47.646622 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.646463 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb"} err="failed to get container status \"0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb\": rpc error: code = NotFound desc = could not find container \"0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb\": container with ID starting with 0beac79bfd8f5099b92b351d5cbc2c7f2428ba585c46dbfb4efb3635fb1cd0eb not found: ID does not exist" Nov 03 09:37:47.804823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.804791 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr"] Nov 03 09:37:47.855024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:47.854981 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5adb63-684c7984f7-ndqmr"] Nov 03 09:37:48.348413 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:48.348367 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:37:49.210138 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:49.210099 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00af5a6c-af44-4a21-b197-c25b51ece5df" path="/var/lib/kubelet/pods/00af5a6c-af44-4a21-b197-c25b51ece5df/volumes" Nov 03 09:37:58.348969 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:37:58.348926 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:38:08.348814 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:08.348774 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:38:15.682569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.682519 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs"] Nov 03 09:38:15.683206 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.683190 5015 cpu_manager.go:414] "containerMap: removing container" podUID="00af5a6c-af44-4a21-b197-c25b51ece5df" containerName="payload-processing" Nov 03 09:38:15.683254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.683208 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00af5a6c-af44-4a21-b197-c25b51ece5df" containerName="payload-processing" Nov 03 09:38:15.683254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.683227 5015 cpu_manager.go:414] "containerMap: removing container" podUID="74e77e1c-5e86-424c-bf70-6d34265486f9" containerName="maas-api" Nov 03 09:38:15.683254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.683234 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e77e1c-5e86-424c-bf70-6d34265486f9" containerName="maas-api" Nov 03 09:38:15.683349 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.683342 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="00af5a6c-af44-4a21-b197-c25b51ece5df" containerName="payload-processing" Nov 03 09:38:15.683392 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.683352 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e77e1c-5e86-424c-bf70-6d34265486f9" containerName="maas-api" Nov 03 09:38:15.688634 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.688612 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.733667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.733302 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-certs\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.733667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.733388 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-podinfo\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.733667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.733423 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-token\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.733667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.733529 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-envoy\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.733667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.733568 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-socket\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.733667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.733636 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-credential-socket\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.734187 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.733704 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-data\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.734187 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.733806 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68r8l\" (UniqueName: \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-kube-api-access-68r8l\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.734187 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.733867 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istiod-ca-cert\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.755621 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.755580 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs"] Nov 03 09:38:15.798721 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.798681 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ff6461-openshift-default-dockercfg-j6r2c\"" Nov 03 09:38:15.835586 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.835464 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-certs\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.835780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.835609 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-podinfo\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.835780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.835635 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-token\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.835780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.835662 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-envoy\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.835780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.835681 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-socket\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.835780 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.835739 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-credential-socket\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.836079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.835792 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-data\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.836079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.835873 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68r8l\" (UniqueName: \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-kube-api-access-68r8l\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.836079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.835923 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istiod-ca-cert\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.836241 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.836200 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-socket\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.836307 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.836238 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-data\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.836307 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.836265 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-credential-socket\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.836426 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.836351 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-certs\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.836646 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.836624 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istiod-ca-cert\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.838710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.838684 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-envoy\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.838880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.838805 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-podinfo\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.899249 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.896827 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68r8l\" (UniqueName: \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-kube-api-access-68r8l\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:15.899249 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:15.897307 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-token\") pod \"e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:16.013966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:16.013930 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:16.456204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:16.456165 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs"] Nov 03 09:38:16.464878 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:16.464840 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:38:16.464971 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:16.464912 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:38:16.464971 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:16.464942 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:38:16.871449 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:16.871414 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" event={"ID":"a9e18b69-fbfa-4a2e-97bb-485ec732384c","Type":"ContainerStarted","Data":"ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb"} Nov 03 09:38:16.871889 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:16.871487 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" event={"ID":"a9e18b69-fbfa-4a2e-97bb-485ec732384c","Type":"ContainerStarted","Data":"56fcc28d1793a3e6148f50c02e42a1435cee5454ec85a7d5e6761f802870979c"} Nov 03 09:38:16.905237 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:38:16.905126 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:38:16.992698 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:16.992646 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" podStartSLOduration=1.9926317070000001 podStartE2EDuration="1.992631707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:38:16.9903191 +0000 UTC m=+3529.915261316" watchObservedRunningTime="2026-11-03 09:38:16.992631707 +0000 UTC m=+3529.917573924" Nov 03 09:38:17.014085 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:17.014058 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:17.020375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:17.020354 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:17.879083 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:17.879055 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:17.880151 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:17.880131 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:38:18.348997 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:18.348956 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:38:23.198249 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:23.198210 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:38:33.198175 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:33.198134 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:38:43.198345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:43.198305 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:38:46.903969 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:38:46.903926 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:38:53.198741 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:38:53.198700 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:39:03.198179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:03.198141 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:39:13.198574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:13.198531 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:39:16.903010 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:39:16.902974 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:39:18.930576 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:18.930541 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr"] Nov 03 09:39:18.931868 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:18.931838 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:39:23.931929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:23.931883 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" containerID="cri-o://a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1" gracePeriod=25 Nov 03 09:39:24.217806 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.217780 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:39:24.242328 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.242300 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-routing-overlay-volume\") pod \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " Nov 03 09:39:24.242468 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.242404 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-provider-credentials\") pod \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " Nov 03 09:39:24.242569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.242467 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-kube-api-access-tg6zj\" (UniqueName: \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-kube-api-access-tg6zj\") pod \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " Nov 03 09:39:24.242569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.242527 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-plugins-config-volume\") pod \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\" (UID: \"6611aae8-9bf5-44e7-a477-060ac9a7fb86\") " Nov 03 09:39:24.242694 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.242668 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-routing-overlay-volume" pod "6611aae8-9bf5-44e7-a477-060ac9a7fb86" (UID: "6611aae8-9bf5-44e7-a477-060ac9a7fb86"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:39:24.242961 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.242938 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-plugins-config-volume" pod "6611aae8-9bf5-44e7-a477-060ac9a7fb86" (UID: "6611aae8-9bf5-44e7-a477-060ac9a7fb86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:39:24.243106 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.243089 5015 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-routing-overlay-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:39:24.243170 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.243112 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6611aae8-9bf5-44e7-a477-060ac9a7fb86-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:39:24.244675 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.244654 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-provider-credentials" pod "6611aae8-9bf5-44e7-a477-060ac9a7fb86" (UID: "6611aae8-9bf5-44e7-a477-060ac9a7fb86"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:39:24.244786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.244765 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-kube-api-access-tg6zj" pod "6611aae8-9bf5-44e7-a477-060ac9a7fb86" (UID: "6611aae8-9bf5-44e7-a477-060ac9a7fb86"). InnerVolumeSpecName "kube-api-access-tg6zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:39:24.344366 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.344333 5015 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-provider-credentials\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:39:24.344366 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.344367 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg6zj\" (UniqueName: \"kubernetes.io/projected/6611aae8-9bf5-44e7-a477-060ac9a7fb86-kube-api-access-tg6zj\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:39:24.490738 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.490706 5015 generic.go:356] "Generic (PLEG): container finished" podID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerID="a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1" exitCode=0 Nov 03 09:39:24.490927 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.490744 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" event={"ID":"6611aae8-9bf5-44e7-a477-060ac9a7fb86","Type":"ContainerDied","Data":"a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1"} Nov 03 09:39:24.490927 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.490751 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" Nov 03 09:39:24.490927 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.490784 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr" event={"ID":"6611aae8-9bf5-44e7-a477-060ac9a7fb86","Type":"ContainerDied","Data":"852b11b05e9da95e1dbbc8f6f2b9d7135674584b6f085a02dd67fabff224513a"} Nov 03 09:39:24.490927 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.490805 5015 scope.go:122] "RemoveContainer" containerID="a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1" Nov 03 09:39:24.503580 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.503558 5015 scope.go:122] "RemoveContainer" containerID="a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1" Nov 03 09:39:24.503861 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:39:24.503840 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1\": container with ID starting with a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1 not found: ID does not exist" containerID="a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1" Nov 03 09:39:24.503925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.503873 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1"} err="failed to get container status \"a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1\": rpc error: code = NotFound desc = could not find container \"a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1\": container with ID starting with a4385fd6e79612b5107bd7ac28e058f5b77e07c31fecb25f77f04c559cce31a1 not found: ID does not exist" Nov 03 09:39:24.662074 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.662024 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr"] Nov 03 09:39:24.667351 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:24.667325 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f5bd745bd-ssdtr"] Nov 03 09:39:25.205138 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:25.205109 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" path="/var/lib/kubelet/pods/6611aae8-9bf5-44e7-a477-060ac9a7fb86/volumes" Nov 03 09:39:27.892910 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:27.892877 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:39:27.910970 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:27.910947 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:39:31.127962 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.127925 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg"] Nov 03 09:39:31.128616 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.128598 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" Nov 03 09:39:31.128695 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.128618 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" Nov 03 09:39:31.128760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.128747 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6611aae8-9bf5-44e7-a477-060ac9a7fb86" containerName="payload-processing" Nov 03 09:39:31.133587 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.133567 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:39:31.190536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.189812 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg"] Nov 03 09:39:31.230522 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.230456 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv6lk\" (UniqueName: \"kubernetes.io/projected/fab9be80-7a31-49ec-a896-2332bfd34bea-kube-api-access-cv6lk\") pod \"payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg\" (UID: \"fab9be80-7a31-49ec-a896-2332bfd34bea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:39:31.230736 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.230598 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab9be80-7a31-49ec-a896-2332bfd34bea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg\" (UID: \"fab9be80-7a31-49ec-a896-2332bfd34bea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:39:31.235563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.235530 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8204d6\"" Nov 03 09:39:31.235735 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.235678 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8204d6-dockercfg-95q5x\"" Nov 03 09:39:31.332074 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.332026 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv6lk\" (UniqueName: \"kubernetes.io/projected/fab9be80-7a31-49ec-a896-2332bfd34bea-kube-api-access-cv6lk\") pod \"payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg\" (UID: \"fab9be80-7a31-49ec-a896-2332bfd34bea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:39:31.332273 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.332151 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab9be80-7a31-49ec-a896-2332bfd34bea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg\" (UID: \"fab9be80-7a31-49ec-a896-2332bfd34bea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:39:31.333149 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.333122 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab9be80-7a31-49ec-a896-2332bfd34bea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg\" (UID: \"fab9be80-7a31-49ec-a896-2332bfd34bea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:39:31.391934 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.391865 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv6lk\" (UniqueName: \"kubernetes.io/projected/fab9be80-7a31-49ec-a896-2332bfd34bea-kube-api-access-cv6lk\") pod \"payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg\" (UID: \"fab9be80-7a31-49ec-a896-2332bfd34bea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:39:31.448700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.448662 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:39:31.833886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:31.833859 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg"] Nov 03 09:39:31.834532 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:39:31.834500 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab9be80_7a31_49ec_a896_2332bfd34bea.slice/crio-d03e79b080f5946f94c8594505368ef031cfdf865a69024915dd53259f556977 WatchSource:0}: Error finding container d03e79b080f5946f94c8594505368ef031cfdf865a69024915dd53259f556977: Status 404 returned error can't find the container with id d03e79b080f5946f94c8594505368ef031cfdf865a69024915dd53259f556977 Nov 03 09:39:32.568769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:32.568732 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" event={"ID":"fab9be80-7a31-49ec-a896-2332bfd34bea","Type":"ContainerStarted","Data":"a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac"} Nov 03 09:39:32.569177 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:32.568792 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" event={"ID":"fab9be80-7a31-49ec-a896-2332bfd34bea","Type":"ContainerStarted","Data":"d03e79b080f5946f94c8594505368ef031cfdf865a69024915dd53259f556977"} Nov 03 09:39:32.569177 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:32.568835 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:39:32.685962 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:32.685900 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" podStartSLOduration=1.685885996 podStartE2EDuration="1.685885996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:39:32.684405668 +0000 UTC m=+3605.609347904" watchObservedRunningTime="2026-11-03 09:39:32.685885996 +0000 UTC m=+3605.610828212" Nov 03 09:39:41.509814 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.509779 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk"] Nov 03 09:39:41.514181 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.514165 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:41.570216 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.570185 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ff6461-serving-cert\"" Nov 03 09:39:41.575961 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.575931 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk"] Nov 03 09:39:41.635647 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.635616 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1e40c8-02dc-4820-804a-bf1d57338561-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:41.635810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.635659 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp2gl\" (UniqueName: \"kubernetes.io/projected/3a1e40c8-02dc-4820-804a-bf1d57338561-kube-api-access-lp2gl\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:41.635810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.635689 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-metrics-certs\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:41.635810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.635732 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-maas-api-tls\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:41.737037 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.736956 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1e40c8-02dc-4820-804a-bf1d57338561-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:41.737037 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.736998 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp2gl\" (UniqueName: \"kubernetes.io/projected/3a1e40c8-02dc-4820-804a-bf1d57338561-kube-api-access-lp2gl\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:41.737037 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.737026 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-metrics-certs\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:41.737381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.737070 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-maas-api-tls\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:41.737381 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:39:41.737204 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-serving-cert: secret "maas-api-e2e-inf-a-ff6461-serving-cert" not found Nov 03 09:39:41.737381 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:39:41.737288 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-maas-api-tls podName:3a1e40c8-02dc-4820-804a-bf1d57338561 nodeName:}" failed. No retries permitted until 2026-11-03 09:39:42.237269948 +0000 UTC m=+3615.162212161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-maas-api-tls") pod "maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" (UID: "3a1e40c8-02dc-4820-804a-bf1d57338561") : secret "maas-api-e2e-inf-a-ff6461-serving-cert" not found Nov 03 09:39:41.738017 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.737997 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1e40c8-02dc-4820-804a-bf1d57338561-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:41.739821 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.739793 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-metrics-certs\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:41.796963 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:41.796893 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp2gl\" (UniqueName: \"kubernetes.io/projected/3a1e40c8-02dc-4820-804a-bf1d57338561-kube-api-access-lp2gl\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:42.244720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:42.244665 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-maas-api-tls\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:42.247454 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:42.247422 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-maas-api-tls\") pod \"maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:42.428645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:42.428615 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:42.820006 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:42.819975 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk"] Nov 03 09:39:42.820795 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:39:42.820769 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1e40c8_02dc_4820_804a_bf1d57338561.slice/crio-644ee5ef9d35693371998536e23b4f8dacb3d53a76056c6f8039d8e626acbe7a WatchSource:0}: Error finding container 644ee5ef9d35693371998536e23b4f8dacb3d53a76056c6f8039d8e626acbe7a: Status 404 returned error can't find the container with id 644ee5ef9d35693371998536e23b4f8dacb3d53a76056c6f8039d8e626acbe7a Nov 03 09:39:42.822191 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:42.822169 5015 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:39:43.583554 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:43.583528 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:39:43.695355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:43.695329 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" event={"ID":"3a1e40c8-02dc-4820-804a-bf1d57338561","Type":"ContainerStarted","Data":"644ee5ef9d35693371998536e23b4f8dacb3d53a76056c6f8039d8e626acbe7a"} Nov 03 09:39:44.703708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:44.703664 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" event={"ID":"3a1e40c8-02dc-4820-804a-bf1d57338561","Type":"ContainerStarted","Data":"ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f"} Nov 03 09:39:44.704232 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:44.703802 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:39:44.825069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:44.825013 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" podStartSLOduration=2.981239858 podStartE2EDuration="3.824998409s" totalImagesPullingTime="843.758551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:39:44.821869591 +0000 UTC m=+3617.746811828" watchObservedRunningTime="2026-11-03 09:39:44.824998409 +0000 UTC m=+3617.749940625" Nov 03 09:39:46.903000 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:39:46.902957 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:39:50.718062 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:39:50.718032 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:40:01.046756 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.046714 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m"] Nov 03 09:40:01.052255 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.052230 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.147252 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.145815 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-dc1444-openshift-default-dockercfg-gwfp7\"" Nov 03 09:40:01.196031 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.195953 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-podinfo\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.196460 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.196434 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-data\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.196700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.196681 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-socket\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.196866 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.196838 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-envoy\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.196940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.196885 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-credential-socket\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.197009 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.196956 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d7493b-3271-4e42-af4b-98ac3a24a03a-istiod-ca-cert\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.197009 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.196993 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-certs\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.197128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.197064 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g845c\" (UniqueName: \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-kube-api-access-g845c\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.197189 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.197127 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-token\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.210889 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.210838 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m"] Nov 03 09:40:01.298345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.298260 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g845c\" (UniqueName: \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-kube-api-access-g845c\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.298345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.298325 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-token\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.298674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.298375 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-podinfo\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.298674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.298447 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-data\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.298674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.298515 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-socket\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.298674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.298538 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-envoy\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.298674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.298554 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-credential-socket\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.298674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.298622 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d7493b-3271-4e42-af4b-98ac3a24a03a-istiod-ca-cert\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.298674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.298660 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-certs\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.299075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.298981 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-socket\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.299075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.299018 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-data\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.299200 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.299112 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-credential-socket\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.299200 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.299161 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-certs\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.299361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.299330 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d7493b-3271-4e42-af4b-98ac3a24a03a-istiod-ca-cert\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.301534 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.301509 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-podinfo\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.301630 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.301510 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-envoy\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.357760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.357724 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g845c\" (UniqueName: \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-kube-api-access-g845c\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.357936 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.357794 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-token\") pod \"e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.381233 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.381209 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:01.828761 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:40:01.828730 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d7493b_3271_4e42_af4b_98ac3a24a03a.slice/crio-e5ff398c3046f84ec1cb099bd751a048d59c2378cc4a8e6aefc760e98cac6b61 WatchSource:0}: Error finding container e5ff398c3046f84ec1cb099bd751a048d59c2378cc4a8e6aefc760e98cac6b61: Status 404 returned error can't find the container with id e5ff398c3046f84ec1cb099bd751a048d59c2378cc4a8e6aefc760e98cac6b61 Nov 03 09:40:01.829065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.829039 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m"] Nov 03 09:40:01.831414 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.831380 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:40:01.831505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.831448 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:40:01.831505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:01.831499 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:40:02.842754 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:02.842709 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" event={"ID":"00d7493b-3271-4e42-af4b-98ac3a24a03a","Type":"ContainerStarted","Data":"a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17"} Nov 03 09:40:02.843160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:02.842782 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" event={"ID":"00d7493b-3271-4e42-af4b-98ac3a24a03a","Type":"ContainerStarted","Data":"e5ff398c3046f84ec1cb099bd751a048d59c2378cc4a8e6aefc760e98cac6b61"} Nov 03 09:40:03.018104 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:03.018050 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" podStartSLOduration=3.018035184 podStartE2EDuration="3.018035184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:40:03.013761407 +0000 UTC m=+3635.938703624" watchObservedRunningTime="2026-11-03 09:40:03.018035184 +0000 UTC m=+3635.942977391" Nov 03 09:40:03.382180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:03.382140 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:03.388884 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:03.388856 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:03.852458 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:03.852415 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:03.853573 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:03.853552 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:40:16.966205 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:40:16.966158 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:40:17.880657 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:17.880616 5015 patch_prober.go:29] interesting pod/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 03 09:40:17.880884 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:17.880711 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 03 09:40:18.013328 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:18.013292 5015 generic.go:356] "Generic (PLEG): container finished" podID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerID="ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb" exitCode=0 Nov 03 09:40:18.013864 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:18.013371 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" event={"ID":"a9e18b69-fbfa-4a2e-97bb-485ec732384c","Type":"ContainerDied","Data":"ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb"} Nov 03 09:40:18.013962 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:18.013921 5015 scope.go:122] "RemoveContainer" containerID="ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb" Nov 03 09:40:18.015706 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:18.015672 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:40:18.015833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:18.015765 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:40:18.015833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:18.015822 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:40:19.022639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:19.022600 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" event={"ID":"a9e18b69-fbfa-4a2e-97bb-485ec732384c","Type":"ContainerStarted","Data":"42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882"} Nov 03 09:40:20.014164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:20.014128 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:40:20.014344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:20.014310 5015 patch_prober.go:29] interesting pod/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 03 09:40:20.014390 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:20.014356 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 03 09:40:21.014527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:21.014463 5015 patch_prober.go:29] interesting pod/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 03 09:40:21.014947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:21.014557 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 03 09:40:22.014587 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:22.014550 5015 patch_prober.go:29] interesting pod/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 03 09:40:22.014980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:22.014605 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 03 09:40:23.014987 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:23.014948 5015 patch_prober.go:29] interesting pod/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 03 09:40:23.015396 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:23.014999 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 03 09:40:24.014764 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:24.014729 5015 patch_prober.go:29] interesting pod/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 03 09:40:24.014958 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:24.014812 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 03 09:40:25.019739 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:25.019707 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:40:25.020179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:25.020158 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:40:25.020911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:25.020891 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:40:37.682836 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:37.682793 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z"] Nov 03 09:40:37.687172 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:37.687147 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:40:37.742628 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:37.742596 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z"] Nov 03 09:40:37.812964 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:37.812889 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-af514d\"" Nov 03 09:40:37.813766 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:37.813501 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-af514d-dockercfg-rkg87\"" Nov 03 09:40:37.818779 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:37.818741 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hk4h\" (UniqueName: \"kubernetes.io/projected/46818ade-3cc4-42d9-8ab2-b0795d09213a-kube-api-access-8hk4h\") pod \"payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z\" (UID: \"46818ade-3cc4-42d9-8ab2-b0795d09213a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:40:37.818944 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:37.818924 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46818ade-3cc4-42d9-8ab2-b0795d09213a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z\" (UID: \"46818ade-3cc4-42d9-8ab2-b0795d09213a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:40:37.920185 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:37.920148 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hk4h\" (UniqueName: \"kubernetes.io/projected/46818ade-3cc4-42d9-8ab2-b0795d09213a-kube-api-access-8hk4h\") pod \"payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z\" (UID: \"46818ade-3cc4-42d9-8ab2-b0795d09213a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:40:37.920368 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:37.920266 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46818ade-3cc4-42d9-8ab2-b0795d09213a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z\" (UID: \"46818ade-3cc4-42d9-8ab2-b0795d09213a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:40:37.921054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:37.921032 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46818ade-3cc4-42d9-8ab2-b0795d09213a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z\" (UID: \"46818ade-3cc4-42d9-8ab2-b0795d09213a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:40:37.979756 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:37.979728 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hk4h\" (UniqueName: \"kubernetes.io/projected/46818ade-3cc4-42d9-8ab2-b0795d09213a-kube-api-access-8hk4h\") pod \"payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z\" (UID: \"46818ade-3cc4-42d9-8ab2-b0795d09213a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:40:38.001109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:38.001084 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:40:38.388660 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:38.388618 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z"] Nov 03 09:40:38.389675 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:40:38.389634 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46818ade_3cc4_42d9_8ab2_b0795d09213a.slice/crio-ba96e0485693b98704506a80c69549123336e260ba3e6094d0b3a19eb2f205b3 WatchSource:0}: Error finding container ba96e0485693b98704506a80c69549123336e260ba3e6094d0b3a19eb2f205b3: Status 404 returned error can't find the container with id ba96e0485693b98704506a80c69549123336e260ba3e6094d0b3a19eb2f205b3 Nov 03 09:40:39.228373 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:39.228339 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" event={"ID":"46818ade-3cc4-42d9-8ab2-b0795d09213a","Type":"ContainerStarted","Data":"1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4"} Nov 03 09:40:39.228789 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:39.228393 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" event={"ID":"46818ade-3cc4-42d9-8ab2-b0795d09213a","Type":"ContainerStarted","Data":"ba96e0485693b98704506a80c69549123336e260ba3e6094d0b3a19eb2f205b3"} Nov 03 09:40:39.228789 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:39.228444 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:40:39.348368 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:39.348298 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" podStartSLOduration=2.348282508 podStartE2EDuration="2.348282508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:40:39.346805426 +0000 UTC m=+3672.271747645" watchObservedRunningTime="2026-11-03 09:40:39.348282508 +0000 UTC m=+3672.273224730" Nov 03 09:40:46.902806 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:40:46.902768 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:40:50.239601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:40:50.239575 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:41:16.904258 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:41:16.904216 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:41:40.599813 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.599729 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq"] Nov 03 09:41:40.604300 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.604280 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:40.662603 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.662570 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq"] Nov 03 09:41:40.707643 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.707615 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-dc1444-serving-cert\"" Nov 03 09:41:40.774848 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.774793 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-metrics-certs\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:40.775035 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.774867 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da81146c-b155-48d6-a6a0-b32fc328dfbe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:40.775082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.775057 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc76l\" (UniqueName: \"kubernetes.io/projected/da81146c-b155-48d6-a6a0-b32fc328dfbe-kube-api-access-cc76l\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:40.775120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.775090 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:40.875990 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.875896 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da81146c-b155-48d6-a6a0-b32fc328dfbe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:40.876168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.876095 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc76l\" (UniqueName: \"kubernetes.io/projected/da81146c-b155-48d6-a6a0-b32fc328dfbe-kube-api-access-cc76l\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:40.876168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.876132 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:40.876292 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.876221 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-metrics-certs\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:40.876383 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:41:40.876359 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-serving-cert: secret "maas-api-e2e-extmod-b-dc1444-serving-cert" not found Nov 03 09:41:40.876469 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:41:40.876457 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-maas-api-tls podName:da81146c-b155-48d6-a6a0-b32fc328dfbe nodeName:}" failed. No retries permitted until 2026-11-03 09:41:41.376434879 +0000 UTC m=+3734.301377077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-maas-api-tls") pod "maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" (UID: "da81146c-b155-48d6-a6a0-b32fc328dfbe") : secret "maas-api-e2e-extmod-b-dc1444-serving-cert" not found Nov 03 09:41:40.876756 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.876739 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da81146c-b155-48d6-a6a0-b32fc328dfbe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:40.879043 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.879015 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-metrics-certs\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:40.938112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:40.938084 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc76l\" (UniqueName: \"kubernetes.io/projected/da81146c-b155-48d6-a6a0-b32fc328dfbe-kube-api-access-cc76l\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:41.384141 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:41.384109 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:41.386966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:41.386940 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc1444-df469d798-btzxq\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:41.518731 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:41.518700 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:41.899667 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:41:41.899638 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda81146c_b155_48d6_a6a0_b32fc328dfbe.slice/crio-b2f83b1f24b02ae3cd0ef4df70bb430b23e5baa9420f2649eecd48e7641bd168 WatchSource:0}: Error finding container b2f83b1f24b02ae3cd0ef4df70bb430b23e5baa9420f2649eecd48e7641bd168: Status 404 returned error can't find the container with id b2f83b1f24b02ae3cd0ef4df70bb430b23e5baa9420f2649eecd48e7641bd168 Nov 03 09:41:41.900056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:41.900034 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq"] Nov 03 09:41:42.756079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:42.756043 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" event={"ID":"da81146c-b155-48d6-a6a0-b32fc328dfbe","Type":"ContainerStarted","Data":"b2f83b1f24b02ae3cd0ef4df70bb430b23e5baa9420f2649eecd48e7641bd168"} Nov 03 09:41:43.765445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:43.765408 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" event={"ID":"da81146c-b155-48d6-a6a0-b32fc328dfbe","Type":"ContainerStarted","Data":"76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615"} Nov 03 09:41:43.765965 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:43.765639 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:41:43.903720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:43.903665 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" podStartSLOduration=3.093538974 podStartE2EDuration="3.903648753s" totalImagesPullingTime="810.109779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:41:43.902271834 +0000 UTC m=+3736.827214069" watchObservedRunningTime="2026-11-03 09:41:43.903648753 +0000 UTC m=+3736.828590966" Nov 03 09:41:46.904230 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:41:46.904187 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:41:49.780469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:41:49.780441 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:42:11.019677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.019645 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz"] Nov 03 09:42:11.034585 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.034550 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.089462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.089431 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-af514d-serving-cert\"" Nov 03 09:42:11.121401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.121366 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-maas-api-tls\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.121401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.121419 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsccv\" (UniqueName: \"kubernetes.io/projected/dc96794c-ba60-4823-8aac-9bbd472cff0e-kube-api-access-vsccv\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.121401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.121641 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-metrics-certs\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.121401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.121802 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc96794c-ba60-4823-8aac-9bbd472cff0e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.145707 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.145671 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz"] Nov 03 09:42:11.223765 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.223731 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc96794c-ba60-4823-8aac-9bbd472cff0e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.223939 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.223866 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-maas-api-tls\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.223939 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.223902 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsccv\" (UniqueName: \"kubernetes.io/projected/dc96794c-ba60-4823-8aac-9bbd472cff0e-kube-api-access-vsccv\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.224049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.224000 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-metrics-certs\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.224049 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:42:11.224021 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-serving-cert: secret "maas-api-e2e-inf-b-af514d-serving-cert" not found Nov 03 09:42:11.224153 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:42:11.224075 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-maas-api-tls podName:dc96794c-ba60-4823-8aac-9bbd472cff0e nodeName:}" failed. No retries permitted until 2026-11-03 09:42:11.724060173 +0000 UTC m=+3764.649002368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-maas-api-tls") pod "maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" (UID: "dc96794c-ba60-4823-8aac-9bbd472cff0e") : secret "maas-api-e2e-inf-b-af514d-serving-cert" not found Nov 03 09:42:11.224421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.224401 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc96794c-ba60-4823-8aac-9bbd472cff0e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.226663 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.226636 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-metrics-certs\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.283809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.283733 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsccv\" (UniqueName: \"kubernetes.io/projected/dc96794c-ba60-4823-8aac-9bbd472cff0e-kube-api-access-vsccv\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.729716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.729674 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-maas-api-tls\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.732351 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.732320 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-maas-api-tls\") pod \"maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:11.953680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:11.953640 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:12.349090 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:12.349007 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz"] Nov 03 09:42:12.359750 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:42:12.359713 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc96794c_ba60_4823_8aac_9bbd472cff0e.slice/crio-efed1c606db149dacb3fdb0d6c124886836c19ce1af1e5b7fc13700aca3f710b WatchSource:0}: Error finding container efed1c606db149dacb3fdb0d6c124886836c19ce1af1e5b7fc13700aca3f710b: Status 404 returned error can't find the container with id efed1c606db149dacb3fdb0d6c124886836c19ce1af1e5b7fc13700aca3f710b Nov 03 09:42:13.049734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:13.049691 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" event={"ID":"dc96794c-ba60-4823-8aac-9bbd472cff0e","Type":"ContainerStarted","Data":"efed1c606db149dacb3fdb0d6c124886836c19ce1af1e5b7fc13700aca3f710b"} Nov 03 09:42:14.065119 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:14.063884 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" event={"ID":"dc96794c-ba60-4823-8aac-9bbd472cff0e","Type":"ContainerStarted","Data":"36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e"} Nov 03 09:42:14.065119 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:14.065066 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:14.186076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:14.185329 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" podStartSLOduration=3.114936356 podStartE2EDuration="4.185307895s" totalImagesPullingTime="1.070371539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:42:14.181739257 +0000 UTC m=+3767.106681475" watchObservedRunningTime="2026-11-03 09:42:14.185307895 +0000 UTC m=+3767.110250112" Nov 03 09:42:15.076303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:15.076269 5015 generic.go:356] "Generic (PLEG): container finished" podID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerID="a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17" exitCode=0 Nov 03 09:42:15.076823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:15.076348 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" event={"ID":"00d7493b-3271-4e42-af4b-98ac3a24a03a","Type":"ContainerDied","Data":"a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17"} Nov 03 09:42:15.076957 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:15.076938 5015 scope.go:122] "RemoveContainer" containerID="a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17" Nov 03 09:42:15.078710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:15.078677 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:42:15.078825 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:15.078747 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:42:15.078825 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:15.078778 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:42:16.089163 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:16.089120 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" event={"ID":"00d7493b-3271-4e42-af4b-98ac3a24a03a","Type":"ContainerStarted","Data":"e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e"} Nov 03 09:42:16.382556 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:16.382114 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:42:16.384437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:16.383369 5015 patch_prober.go:29] interesting pod/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 09:42:16.384437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:16.383512 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 09:42:16.904633 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:42:16.904584 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:42:17.381563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:17.381523 5015 patch_prober.go:29] interesting pod/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 09:42:17.382142 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:17.381596 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 09:42:18.382238 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:18.382202 5015 patch_prober.go:29] interesting pod/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 09:42:18.382661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:18.382263 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 09:42:18.860617 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:18.860577 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:42:19.382093 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:19.382057 5015 patch_prober.go:29] interesting pod/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 09:42:19.382288 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:19.382115 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 09:42:20.381916 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:20.381876 5015 patch_prober.go:29] interesting pod/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 09:42:20.382111 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:20.381939 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 09:42:21.095508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:21.095451 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:42:21.386246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:21.386166 5015 patch_prober.go:29] interesting pod/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 09:42:21.386553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:21.386513 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 09:42:22.387150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:22.387120 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:42:22.388314 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:22.388293 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:42:34.714708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:34.714673 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg"] Nov 03 09:42:39.715829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:39.715782 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" podUID="fab9be80-7a31-49ec-a896-2332bfd34bea" containerName="payload-pre-processing" containerID="cri-o://a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac" gracePeriod=25 Nov 03 09:42:39.997131 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:39.997101 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:42:40.170279 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.170243 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab9be80-7a31-49ec-a896-2332bfd34bea-kube-api-access-cv6lk\" (UniqueName: \"kubernetes.io/projected/fab9be80-7a31-49ec-a896-2332bfd34bea-kube-api-access-cv6lk\") pod \"fab9be80-7a31-49ec-a896-2332bfd34bea\" (UID: \"fab9be80-7a31-49ec-a896-2332bfd34bea\") " Nov 03 09:42:40.170467 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.170337 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab9be80-7a31-49ec-a896-2332bfd34bea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab9be80-7a31-49ec-a896-2332bfd34bea-plugins-config-volume\") pod \"fab9be80-7a31-49ec-a896-2332bfd34bea\" (UID: \"fab9be80-7a31-49ec-a896-2332bfd34bea\") " Nov 03 09:42:40.170788 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.170762 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab9be80-7a31-49ec-a896-2332bfd34bea-plugins-config-volume" pod "fab9be80-7a31-49ec-a896-2332bfd34bea" (UID: "fab9be80-7a31-49ec-a896-2332bfd34bea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:42:40.170895 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.170882 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab9be80-7a31-49ec-a896-2332bfd34bea-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:40.172728 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.172704 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab9be80-7a31-49ec-a896-2332bfd34bea-kube-api-access-cv6lk" pod "fab9be80-7a31-49ec-a896-2332bfd34bea" (UID: "fab9be80-7a31-49ec-a896-2332bfd34bea"). InnerVolumeSpecName "kube-api-access-cv6lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:42:40.272466 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.272413 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv6lk\" (UniqueName: \"kubernetes.io/projected/fab9be80-7a31-49ec-a896-2332bfd34bea-kube-api-access-cv6lk\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:40.303069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.303034 5015 generic.go:356] "Generic (PLEG): container finished" podID="fab9be80-7a31-49ec-a896-2332bfd34bea" containerID="a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac" exitCode=0 Nov 03 09:42:40.303243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.303075 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" Nov 03 09:42:40.303243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.303112 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" event={"ID":"fab9be80-7a31-49ec-a896-2332bfd34bea","Type":"ContainerDied","Data":"a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac"} Nov 03 09:42:40.303243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.303166 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg" event={"ID":"fab9be80-7a31-49ec-a896-2332bfd34bea","Type":"ContainerDied","Data":"d03e79b080f5946f94c8594505368ef031cfdf865a69024915dd53259f556977"} Nov 03 09:42:40.303243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.303188 5015 scope.go:122] "RemoveContainer" containerID="a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac" Nov 03 09:42:40.319742 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.319714 5015 scope.go:122] "RemoveContainer" containerID="a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac" Nov 03 09:42:40.320054 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:42:40.320031 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac\": container with ID starting with a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac not found: ID does not exist" containerID="a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac" Nov 03 09:42:40.320113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.320067 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac"} err="failed to get container status \"a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac\": rpc error: code = NotFound desc = could not find container \"a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac\": container with ID starting with a0976c26abbc0bafd76a71e278d81d35d6ba9d443843beed8dce66a43c9962ac not found: ID does not exist" Nov 03 09:42:40.474887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.474855 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg"] Nov 03 09:42:40.479698 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:40.479673 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8204d6-b574cbb55-vshzg"] Nov 03 09:42:41.206046 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:41.206005 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab9be80-7a31-49ec-a896-2332bfd34bea" path="/var/lib/kubelet/pods/fab9be80-7a31-49ec-a896-2332bfd34bea/volumes" Nov 03 09:42:46.905667 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:42:46.905626 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:42:52.395980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.395945 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk"] Nov 03 09:42:52.396434 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.396165 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" podUID="3a1e40c8-02dc-4820-804a-bf1d57338561" containerName="maas-api" containerID="cri-o://ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f" gracePeriod=30 Nov 03 09:42:52.726751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.726719 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:42:52.834947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.834914 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-maas-api-tls\") pod \"3a1e40c8-02dc-4820-804a-bf1d57338561\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " Nov 03 09:42:52.834947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.834949 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-metrics-certs\") pod \"3a1e40c8-02dc-4820-804a-bf1d57338561\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " Nov 03 09:42:52.835330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.835006 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1e40c8-02dc-4820-804a-bf1d57338561-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1e40c8-02dc-4820-804a-bf1d57338561-openshift-service-ca\") pod \"3a1e40c8-02dc-4820-804a-bf1d57338561\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " Nov 03 09:42:52.835330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.835272 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1e40c8-02dc-4820-804a-bf1d57338561-kube-api-access-lp2gl\" (UniqueName: \"kubernetes.io/projected/3a1e40c8-02dc-4820-804a-bf1d57338561-kube-api-access-lp2gl\") pod \"3a1e40c8-02dc-4820-804a-bf1d57338561\" (UID: \"3a1e40c8-02dc-4820-804a-bf1d57338561\") " Nov 03 09:42:52.835557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.835529 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1e40c8-02dc-4820-804a-bf1d57338561-openshift-service-ca" pod "3a1e40c8-02dc-4820-804a-bf1d57338561" (UID: "3a1e40c8-02dc-4820-804a-bf1d57338561"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:42:52.835944 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.835908 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1e40c8-02dc-4820-804a-bf1d57338561-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:52.837688 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.837658 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-maas-api-tls" pod "3a1e40c8-02dc-4820-804a-bf1d57338561" (UID: "3a1e40c8-02dc-4820-804a-bf1d57338561"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:42:52.838240 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.838204 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1e40c8-02dc-4820-804a-bf1d57338561-kube-api-access-lp2gl" pod "3a1e40c8-02dc-4820-804a-bf1d57338561" (UID: "3a1e40c8-02dc-4820-804a-bf1d57338561"). InnerVolumeSpecName "kube-api-access-lp2gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:42:52.842279 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.842251 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-metrics-certs" pod "3a1e40c8-02dc-4820-804a-bf1d57338561" (UID: "3a1e40c8-02dc-4820-804a-bf1d57338561"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:42:52.937699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.937604 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp2gl\" (UniqueName: \"kubernetes.io/projected/3a1e40c8-02dc-4820-804a-bf1d57338561-kube-api-access-lp2gl\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:52.937699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.937647 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:52.937699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:52.937661 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a1e40c8-02dc-4820-804a-bf1d57338561-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:53.446511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:53.446437 5015 generic.go:356] "Generic (PLEG): container finished" podID="3a1e40c8-02dc-4820-804a-bf1d57338561" containerID="ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f" exitCode=0 Nov 03 09:42:53.447024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:53.446649 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" event={"ID":"3a1e40c8-02dc-4820-804a-bf1d57338561","Type":"ContainerDied","Data":"ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f"} Nov 03 09:42:53.447024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:53.446708 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" event={"ID":"3a1e40c8-02dc-4820-804a-bf1d57338561","Type":"ContainerDied","Data":"644ee5ef9d35693371998536e23b4f8dacb3d53a76056c6f8039d8e626acbe7a"} Nov 03 09:42:53.447024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:53.446738 5015 scope.go:122] "RemoveContainer" containerID="ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f" Nov 03 09:42:53.447024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:53.446893 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk" Nov 03 09:42:53.465765 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:53.465738 5015 scope.go:122] "RemoveContainer" containerID="ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f" Nov 03 09:42:53.466149 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:42:53.466118 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f\": container with ID starting with ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f not found: ID does not exist" containerID="ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f" Nov 03 09:42:53.466274 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:53.466160 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f"} err="failed to get container status \"ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f\": rpc error: code = NotFound desc = could not find container \"ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f\": container with ID starting with ea00d838dffa782e1d49367ec9fba35d6a8ddbe3ae309a0e7c150e517534cd9f not found: ID does not exist" Nov 03 09:42:53.618899 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:53.618864 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk"] Nov 03 09:42:53.669031 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:53.668988 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff6461-7fc76c7b84-tzdwk"] Nov 03 09:42:55.205357 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:55.205322 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1e40c8-02dc-4820-804a-bf1d57338561" path="/var/lib/kubelet/pods/3a1e40c8-02dc-4820-804a-bf1d57338561/volumes" Nov 03 09:42:55.836280 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:55.836243 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs"] Nov 03 09:42:55.836575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:42:55.836457 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" containerID="cri-o://42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882" gracePeriod=30 Nov 03 09:43:01.116028 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.115956 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:43:01.258220 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258177 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-socket\") pod \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " Nov 03 09:43:01.258410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258241 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-certs\") pod \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " Nov 03 09:43:01.258410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258314 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-credential-socket\") pod \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " Nov 03 09:43:01.258410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258337 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-envoy\") pod \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " Nov 03 09:43:01.258410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258405 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-data\") pod \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " Nov 03 09:43:01.258680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258439 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-podinfo\") pod \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " Nov 03 09:43:01.258680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258464 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-token\" (UniqueName: \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-token\") pod \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " Nov 03 09:43:01.258680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258569 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istiod-ca-cert\") pod \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " Nov 03 09:43:01.258680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258626 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-kube-api-access-68r8l\" (UniqueName: \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-kube-api-access-68r8l\") pod \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\" (UID: \"a9e18b69-fbfa-4a2e-97bb-485ec732384c\") " Nov 03 09:43:01.258680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258561 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-socket" pod "a9e18b69-fbfa-4a2e-97bb-485ec732384c" (UID: "a9e18b69-fbfa-4a2e-97bb-485ec732384c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:43:01.258680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258638 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-certs" pod "a9e18b69-fbfa-4a2e-97bb-485ec732384c" (UID: "a9e18b69-fbfa-4a2e-97bb-485ec732384c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:43:01.258980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258743 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-credential-socket" pod "a9e18b69-fbfa-4a2e-97bb-485ec732384c" (UID: "a9e18b69-fbfa-4a2e-97bb-485ec732384c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:43:01.258980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.258763 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-data" pod "a9e18b69-fbfa-4a2e-97bb-485ec732384c" (UID: "a9e18b69-fbfa-4a2e-97bb-485ec732384c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:43:01.259140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.259116 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istiod-ca-cert" pod "a9e18b69-fbfa-4a2e-97bb-485ec732384c" (UID: "a9e18b69-fbfa-4a2e-97bb-485ec732384c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:43:01.259202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.259154 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:43:01.259202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.259175 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:43:01.259202 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.259188 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:43:01.259331 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.259201 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:43:01.261044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.261015 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-podinfo" pod "a9e18b69-fbfa-4a2e-97bb-485ec732384c" (UID: "a9e18b69-fbfa-4a2e-97bb-485ec732384c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:43:01.261237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.261217 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-kube-api-access-68r8l" pod "a9e18b69-fbfa-4a2e-97bb-485ec732384c" (UID: "a9e18b69-fbfa-4a2e-97bb-485ec732384c"). InnerVolumeSpecName "kube-api-access-68r8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:43:01.261445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.261428 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-envoy" pod "a9e18b69-fbfa-4a2e-97bb-485ec732384c" (UID: "a9e18b69-fbfa-4a2e-97bb-485ec732384c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:43:01.261445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.261431 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-token" pod "a9e18b69-fbfa-4a2e-97bb-485ec732384c" (UID: "a9e18b69-fbfa-4a2e-97bb-485ec732384c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:43:01.360489 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.360430 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:43:01.360489 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.360470 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:43:01.360752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.360505 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:43:01.360752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.360519 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68r8l\" (UniqueName: \"kubernetes.io/projected/a9e18b69-fbfa-4a2e-97bb-485ec732384c-kube-api-access-68r8l\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:43:01.360752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.360536 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e18b69-fbfa-4a2e-97bb-485ec732384c-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:43:01.517999 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.517954 5015 generic.go:356] "Generic (PLEG): container finished" podID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerID="42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882" exitCode=0 Nov 03 09:43:01.518214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.518013 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" Nov 03 09:43:01.518214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.518036 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" event={"ID":"a9e18b69-fbfa-4a2e-97bb-485ec732384c","Type":"ContainerDied","Data":"42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882"} Nov 03 09:43:01.518214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.518095 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs" event={"ID":"a9e18b69-fbfa-4a2e-97bb-485ec732384c","Type":"ContainerDied","Data":"56fcc28d1793a3e6148f50c02e42a1435cee5454ec85a7d5e6761f802870979c"} Nov 03 09:43:01.518214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.518116 5015 scope.go:122] "RemoveContainer" containerID="42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882" Nov 03 09:43:01.530748 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.530728 5015 scope.go:122] "RemoveContainer" containerID="ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb" Nov 03 09:43:01.544993 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.544968 5015 scope.go:122] "RemoveContainer" containerID="42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882" Nov 03 09:43:01.545382 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:43:01.545358 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882\": container with ID starting with 42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882 not found: ID does not exist" containerID="42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882" Nov 03 09:43:01.545509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.545391 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882"} err="failed to get container status \"42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882\": rpc error: code = NotFound desc = could not find container \"42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882\": container with ID starting with 42ba4d3a92912164cd6d65da53609e6748044e9f7f2aa2e77320c1579fcb7882 not found: ID does not exist" Nov 03 09:43:01.545509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.545411 5015 scope.go:122] "RemoveContainer" containerID="ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb" Nov 03 09:43:01.545781 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:43:01.545763 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb\": container with ID starting with ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb not found: ID does not exist" containerID="ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb" Nov 03 09:43:01.545831 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.545788 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb"} err="failed to get container status \"ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb\": rpc error: code = NotFound desc = could not find container \"ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb\": container with ID starting with ea6877162d8730a623212cd32a1c24132160a3b105a7fb7aaaf3b719579ba3fb not found: ID does not exist" Nov 03 09:43:01.695510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.695461 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs"] Nov 03 09:43:01.743658 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:01.743628 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ff6461-openshift-default-6bc9797957-g5rfs"] Nov 03 09:43:03.207982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:43:03.207946 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" path="/var/lib/kubelet/pods/a9e18b69-fbfa-4a2e-97bb-485ec732384c/volumes" Nov 03 09:43:16.921734 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:43:16.921622 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:43:46.903280 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:43:46.903241 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:44:16.904756 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:44:16.904706 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:44:27.949711 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:27.949683 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:44:27.968727 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:27.968703 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:44:43.813601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:43.813508 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq"] Nov 03 09:44:43.814010 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:43.813746 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" podUID="da81146c-b155-48d6-a6a0-b32fc328dfbe" containerName="maas-api" containerID="cri-o://76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615" gracePeriod=30 Nov 03 09:44:44.156852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.156819 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:44:44.285383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.282194 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da81146c-b155-48d6-a6a0-b32fc328dfbe-kube-api-access-cc76l\" (UniqueName: \"kubernetes.io/projected/da81146c-b155-48d6-a6a0-b32fc328dfbe-kube-api-access-cc76l\") pod \"da81146c-b155-48d6-a6a0-b32fc328dfbe\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " Nov 03 09:44:44.285383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.282283 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-metrics-certs\") pod \"da81146c-b155-48d6-a6a0-b32fc328dfbe\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " Nov 03 09:44:44.285383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.282318 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-maas-api-tls\") pod \"da81146c-b155-48d6-a6a0-b32fc328dfbe\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " Nov 03 09:44:44.285383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.282410 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da81146c-b155-48d6-a6a0-b32fc328dfbe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da81146c-b155-48d6-a6a0-b32fc328dfbe-openshift-service-ca\") pod \"da81146c-b155-48d6-a6a0-b32fc328dfbe\" (UID: \"da81146c-b155-48d6-a6a0-b32fc328dfbe\") " Nov 03 09:44:44.287167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.287130 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da81146c-b155-48d6-a6a0-b32fc328dfbe-openshift-service-ca" pod "da81146c-b155-48d6-a6a0-b32fc328dfbe" (UID: "da81146c-b155-48d6-a6a0-b32fc328dfbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:44:44.301493 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.298803 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-maas-api-tls" pod "da81146c-b155-48d6-a6a0-b32fc328dfbe" (UID: "da81146c-b155-48d6-a6a0-b32fc328dfbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:44:44.301493 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.298939 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-metrics-certs" pod "da81146c-b155-48d6-a6a0-b32fc328dfbe" (UID: "da81146c-b155-48d6-a6a0-b32fc328dfbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:44:44.305975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.305653 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da81146c-b155-48d6-a6a0-b32fc328dfbe-kube-api-access-cc76l" pod "da81146c-b155-48d6-a6a0-b32fc328dfbe" (UID: "da81146c-b155-48d6-a6a0-b32fc328dfbe"). InnerVolumeSpecName "kube-api-access-cc76l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:44:44.384653 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.384564 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc76l\" (UniqueName: \"kubernetes.io/projected/da81146c-b155-48d6-a6a0-b32fc328dfbe-kube-api-access-cc76l\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:44.384653 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.384597 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:44.384653 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.384607 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da81146c-b155-48d6-a6a0-b32fc328dfbe-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:44.384653 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.384617 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da81146c-b155-48d6-a6a0-b32fc328dfbe-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:44.466060 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.466002 5015 generic.go:356] "Generic (PLEG): container finished" podID="da81146c-b155-48d6-a6a0-b32fc328dfbe" containerID="76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615" exitCode=0 Nov 03 09:44:44.466578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.466350 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" Nov 03 09:44:44.466578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.466373 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" event={"ID":"da81146c-b155-48d6-a6a0-b32fc328dfbe","Type":"ContainerDied","Data":"76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615"} Nov 03 09:44:44.466578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.466446 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq" event={"ID":"da81146c-b155-48d6-a6a0-b32fc328dfbe","Type":"ContainerDied","Data":"b2f83b1f24b02ae3cd0ef4df70bb430b23e5baa9420f2649eecd48e7641bd168"} Nov 03 09:44:44.466578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.466523 5015 scope.go:122] "RemoveContainer" containerID="76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615" Nov 03 09:44:44.481379 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.481353 5015 scope.go:122] "RemoveContainer" containerID="76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615" Nov 03 09:44:44.481796 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:44:44.481772 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615\": container with ID starting with 76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615 not found: ID does not exist" containerID="76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615" Nov 03 09:44:44.481887 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.481809 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615"} err="failed to get container status \"76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615\": rpc error: code = NotFound desc = could not find container \"76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615\": container with ID starting with 76beda5d88e2aa968da0c3d30114c9d8002b7b4b28a4976576ff4b4764e6a615 not found: ID does not exist" Nov 03 09:44:44.639586 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.639488 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq"] Nov 03 09:44:44.644948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:44.644916 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc1444-df469d798-btzxq"] Nov 03 09:44:45.236512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:45.233193 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da81146c-b155-48d6-a6a0-b32fc328dfbe" path="/var/lib/kubelet/pods/da81146c-b155-48d6-a6a0-b32fc328dfbe/volumes" Nov 03 09:44:46.919971 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:44:46.919911 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:44:57.176190 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:57.176154 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m"] Nov 03 09:44:57.176758 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:44:57.176413 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" containerID="cri-o://e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e" gracePeriod=30 Nov 03 09:45:00.193049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193011 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w"] Nov 03 09:45:00.193761 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193744 5015 cpu_manager.go:414] "containerMap: removing container" podUID="fab9be80-7a31-49ec-a896-2332bfd34bea" containerName="payload-pre-processing" Nov 03 09:45:00.193829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193763 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab9be80-7a31-49ec-a896-2332bfd34bea" containerName="payload-pre-processing" Nov 03 09:45:00.193829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193778 5015 cpu_manager.go:414] "containerMap: removing container" podUID="3a1e40c8-02dc-4820-804a-bf1d57338561" containerName="maas-api" Nov 03 09:45:00.193829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193784 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1e40c8-02dc-4820-804a-bf1d57338561" containerName="maas-api" Nov 03 09:45:00.193829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193798 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" Nov 03 09:45:00.193829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193803 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" Nov 03 09:45:00.193829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193813 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" Nov 03 09:45:00.193829 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193818 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" Nov 03 09:45:00.194055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193849 5015 cpu_manager.go:414] "containerMap: removing container" podUID="da81146c-b155-48d6-a6a0-b32fc328dfbe" containerName="maas-api" Nov 03 09:45:00.194055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193855 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da81146c-b155-48d6-a6a0-b32fc328dfbe" containerName="maas-api" Nov 03 09:45:00.194055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193951 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab9be80-7a31-49ec-a896-2332bfd34bea" containerName="payload-pre-processing" Nov 03 09:45:00.194055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193960 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1e40c8-02dc-4820-804a-bf1d57338561" containerName="maas-api" Nov 03 09:45:00.194055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193968 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" Nov 03 09:45:00.194055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193973 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e18b69-fbfa-4a2e-97bb-485ec732384c" containerName="istio-proxy" Nov 03 09:45:00.194055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.193984 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="da81146c-b155-48d6-a6a0-b32fc328dfbe" containerName="maas-api" Nov 03 09:45:00.197575 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.197560 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" Nov 03 09:45:00.213859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.213831 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7"] Nov 03 09:45:00.217983 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.217962 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" Nov 03 09:45:00.219198 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.219176 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9"] Nov 03 09:45:00.223037 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.223015 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" Nov 03 09:45:00.296759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.296683 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7642521-b6b0-4461-8dc4-da4e95811bb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w\" (UID: \"a7642521-b6b0-4461-8dc4-da4e95811bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" Nov 03 09:45:00.297102 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.297083 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thc4s\" (UniqueName: \"kubernetes.io/projected/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-kube-api-access-thc4s\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7\" (UID: \"746c99f8-0c99-4eb2-a8a0-4868089ee5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" Nov 03 09:45:00.297160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.297120 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7\" (UID: \"746c99f8-0c99-4eb2-a8a0-4868089ee5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" Nov 03 09:45:00.297210 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.297197 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htsr5\" (UniqueName: \"kubernetes.io/projected/a7642521-b6b0-4461-8dc4-da4e95811bb9-kube-api-access-htsr5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w\" (UID: \"a7642521-b6b0-4461-8dc4-da4e95811bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" Nov 03 09:45:00.326044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.326003 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w"] Nov 03 09:45:00.327260 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.327228 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:45:00.375253 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.375216 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9"] Nov 03 09:45:00.381268 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.381230 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7"] Nov 03 09:45:00.398158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.398130 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thc4s\" (UniqueName: \"kubernetes.io/projected/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-kube-api-access-thc4s\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7\" (UID: \"746c99f8-0c99-4eb2-a8a0-4868089ee5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" Nov 03 09:45:00.398311 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.398172 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7\" (UID: \"746c99f8-0c99-4eb2-a8a0-4868089ee5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" Nov 03 09:45:00.398311 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.398255 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htsr5\" (UniqueName: \"kubernetes.io/projected/a7642521-b6b0-4461-8dc4-da4e95811bb9-kube-api-access-htsr5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w\" (UID: \"a7642521-b6b0-4461-8dc4-da4e95811bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" Nov 03 09:45:00.398435 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.398330 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5038b440-677f-4db7-886a-79a1f6c1d965-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9\" (UID: \"5038b440-677f-4db7-886a-79a1f6c1d965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" Nov 03 09:45:00.398545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.398452 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7642521-b6b0-4461-8dc4-da4e95811bb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w\" (UID: \"a7642521-b6b0-4461-8dc4-da4e95811bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" Nov 03 09:45:00.398545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.398508 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zkkk\" (UniqueName: \"kubernetes.io/projected/5038b440-677f-4db7-886a-79a1f6c1d965-kube-api-access-5zkkk\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9\" (UID: \"5038b440-677f-4db7-886a-79a1f6c1d965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" Nov 03 09:45:00.399133 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.399104 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7642521-b6b0-4461-8dc4-da4e95811bb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w\" (UID: \"a7642521-b6b0-4461-8dc4-da4e95811bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" Nov 03 09:45:00.399249 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.399178 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7\" (UID: \"746c99f8-0c99-4eb2-a8a0-4868089ee5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" Nov 03 09:45:00.484528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.483130 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htsr5\" (UniqueName: \"kubernetes.io/projected/a7642521-b6b0-4461-8dc4-da4e95811bb9-kube-api-access-htsr5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w\" (UID: \"a7642521-b6b0-4461-8dc4-da4e95811bb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" Nov 03 09:45:00.484528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.483574 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thc4s\" (UniqueName: \"kubernetes.io/projected/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-kube-api-access-thc4s\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7\" (UID: \"746c99f8-0c99-4eb2-a8a0-4868089ee5c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" Nov 03 09:45:00.499493 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.499435 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zkkk\" (UniqueName: \"kubernetes.io/projected/5038b440-677f-4db7-886a-79a1f6c1d965-kube-api-access-5zkkk\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9\" (UID: \"5038b440-677f-4db7-886a-79a1f6c1d965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" Nov 03 09:45:00.499754 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.499732 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5038b440-677f-4db7-886a-79a1f6c1d965-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9\" (UID: \"5038b440-677f-4db7-886a-79a1f6c1d965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" Nov 03 09:45:00.500428 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.500409 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5038b440-677f-4db7-886a-79a1f6c1d965-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9\" (UID: \"5038b440-677f-4db7-886a-79a1f6c1d965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" Nov 03 09:45:00.513121 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.513103 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" Nov 03 09:45:00.533656 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.533633 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" Nov 03 09:45:00.567743 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.567703 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zkkk\" (UniqueName: \"kubernetes.io/projected/5038b440-677f-4db7-886a-79a1f6c1d965-kube-api-access-5zkkk\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9\" (UID: \"5038b440-677f-4db7-886a-79a1f6c1d965\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" Nov 03 09:45:00.840465 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.840388 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" Nov 03 09:45:00.917529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.915702 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w"] Nov 03 09:45:00.918529 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:45:00.918432 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7642521_b6b0_4461_8dc4_da4e95811bb9.slice/crio-cea9283d8c2ddae6fd9e5221f2edb8d71a9f8326829d55704bb202b4aa6b57ce WatchSource:0}: Error finding container cea9283d8c2ddae6fd9e5221f2edb8d71a9f8326829d55704bb202b4aa6b57ce: Status 404 returned error can't find the container with id cea9283d8c2ddae6fd9e5221f2edb8d71a9f8326829d55704bb202b4aa6b57ce Nov 03 09:45:00.966931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:00.966899 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7"] Nov 03 09:45:01.223980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:01.223940 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9"] Nov 03 09:45:01.224413 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:45:01.224000 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5038b440_677f_4db7_886a_79a1f6c1d965.slice/crio-83e70690d31d54ac2931d010c8f78ebd537b9a273d9c17539ee46d5f65d223f7 WatchSource:0}: Error finding container 83e70690d31d54ac2931d010c8f78ebd537b9a273d9c17539ee46d5f65d223f7: Status 404 returned error can't find the container with id 83e70690d31d54ac2931d010c8f78ebd537b9a273d9c17539ee46d5f65d223f7 Nov 03 09:45:01.655247 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:01.655209 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" event={"ID":"a7642521-b6b0-4461-8dc4-da4e95811bb9","Type":"ContainerStarted","Data":"8f9d171c9911c7fa34f8a93bec5c6f256846ac95732c3482eedf954676eeb355"} Nov 03 09:45:01.655432 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:01.655269 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" event={"ID":"a7642521-b6b0-4461-8dc4-da4e95811bb9","Type":"ContainerStarted","Data":"cea9283d8c2ddae6fd9e5221f2edb8d71a9f8326829d55704bb202b4aa6b57ce"} Nov 03 09:45:01.657336 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:01.657303 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" event={"ID":"746c99f8-0c99-4eb2-a8a0-4868089ee5c1","Type":"ContainerStarted","Data":"8f345f2c86c15e7e98d9df07a9b5c9fde71637d8346047704911111b4b9d131d"} Nov 03 09:45:01.657543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:01.657360 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" event={"ID":"746c99f8-0c99-4eb2-a8a0-4868089ee5c1","Type":"ContainerStarted","Data":"663d9de9dfac7f41dd70a0eef099a753d416d58e1f6c3da124c281423832f9a3"} Nov 03 09:45:01.658835 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:01.658812 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" event={"ID":"5038b440-677f-4db7-886a-79a1f6c1d965","Type":"ContainerStarted","Data":"83e70690d31d54ac2931d010c8f78ebd537b9a273d9c17539ee46d5f65d223f7"} Nov 03 09:45:01.773843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:01.773772 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" podStartSLOduration=1.773752425 podStartE2EDuration="1.773752425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:01.772420031 +0000 UTC m=+3934.697362266" watchObservedRunningTime="2026-11-03 09:45:01.773752425 +0000 UTC m=+3934.698694643" Nov 03 09:45:01.895281 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:01.895222 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" podStartSLOduration=1.895205373 podStartE2EDuration="1.895205373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:01.891577076 +0000 UTC m=+3934.816519313" watchObservedRunningTime="2026-11-03 09:45:01.895205373 +0000 UTC m=+3934.820147589" Nov 03 09:45:02.607952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.607923 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:45:02.671229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.671193 5015 generic.go:356] "Generic (PLEG): container finished" podID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerID="e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e" exitCode=0 Nov 03 09:45:02.671441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.671252 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" Nov 03 09:45:02.671441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.671287 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" event={"ID":"00d7493b-3271-4e42-af4b-98ac3a24a03a","Type":"ContainerDied","Data":"e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e"} Nov 03 09:45:02.671441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.671361 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m" event={"ID":"00d7493b-3271-4e42-af4b-98ac3a24a03a","Type":"ContainerDied","Data":"e5ff398c3046f84ec1cb099bd751a048d59c2378cc4a8e6aefc760e98cac6b61"} Nov 03 09:45:02.671441 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.671394 5015 scope.go:122] "RemoveContainer" containerID="e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e" Nov 03 09:45:02.673392 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.673368 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" event={"ID":"a7642521-b6b0-4461-8dc4-da4e95811bb9","Type":"ContainerDied","Data":"8f9d171c9911c7fa34f8a93bec5c6f256846ac95732c3482eedf954676eeb355"} Nov 03 09:45:02.673392 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.673310 5015 generic.go:356] "Generic (PLEG): container finished" podID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerID="8f9d171c9911c7fa34f8a93bec5c6f256846ac95732c3482eedf954676eeb355" exitCode=60 Nov 03 09:45:02.673749 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.673731 5015 scope.go:122] "RemoveContainer" containerID="8f9d171c9911c7fa34f8a93bec5c6f256846ac95732c3482eedf954676eeb355" Nov 03 09:45:02.675666 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.675645 5015 generic.go:356] "Generic (PLEG): container finished" podID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerID="8f345f2c86c15e7e98d9df07a9b5c9fde71637d8346047704911111b4b9d131d" exitCode=60 Nov 03 09:45:02.675802 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.675733 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" event={"ID":"746c99f8-0c99-4eb2-a8a0-4868089ee5c1","Type":"ContainerDied","Data":"8f345f2c86c15e7e98d9df07a9b5c9fde71637d8346047704911111b4b9d131d"} Nov 03 09:45:02.675951 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.675937 5015 scope.go:122] "RemoveContainer" containerID="8f345f2c86c15e7e98d9df07a9b5c9fde71637d8346047704911111b4b9d131d" Nov 03 09:45:02.679304 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.679286 5015 generic.go:356] "Generic (PLEG): container finished" podID="5038b440-677f-4db7-886a-79a1f6c1d965" containerID="7683795a2631e68c5edff464f2b8f9bb24b6ae6fc5702e21693786618c344814" exitCode=60 Nov 03 09:45:02.679368 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.679332 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" event={"ID":"5038b440-677f-4db7-886a-79a1f6c1d965","Type":"ContainerDied","Data":"7683795a2631e68c5edff464f2b8f9bb24b6ae6fc5702e21693786618c344814"} Nov 03 09:45:02.679661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.679645 5015 scope.go:122] "RemoveContainer" containerID="7683795a2631e68c5edff464f2b8f9bb24b6ae6fc5702e21693786618c344814" Nov 03 09:45:02.693245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.693220 5015 scope.go:122] "RemoveContainer" containerID="a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17" Nov 03 09:45:02.707751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.707719 5015 scope.go:122] "RemoveContainer" containerID="e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e" Nov 03 09:45:02.708122 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:02.708097 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e\": container with ID starting with e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e not found: ID does not exist" containerID="e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e" Nov 03 09:45:02.708199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.708134 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e"} err="failed to get container status \"e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e\": rpc error: code = NotFound desc = could not find container \"e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e\": container with ID starting with e71dabf1191b252290290bce590a1730a4a108770d064dba54cadd356f5ce80e not found: ID does not exist" Nov 03 09:45:02.708199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.708163 5015 scope.go:122] "RemoveContainer" containerID="a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17" Nov 03 09:45:02.708527 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:02.708495 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17\": container with ID starting with a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17 not found: ID does not exist" containerID="a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17" Nov 03 09:45:02.708606 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.708539 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17"} err="failed to get container status \"a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17\": rpc error: code = NotFound desc = could not find container \"a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17\": container with ID starting with a8c644f2b9d95a436efec28c4b93ab31b355df72a5ec1820fefb1eb766022f17 not found: ID does not exist" Nov 03 09:45:02.757298 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757273 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-socket\") pod \"00d7493b-3271-4e42-af4b-98ac3a24a03a\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " Nov 03 09:45:02.757452 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757387 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d7493b-3271-4e42-af4b-98ac3a24a03a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d7493b-3271-4e42-af4b-98ac3a24a03a-istiod-ca-cert\") pod \"00d7493b-3271-4e42-af4b-98ac3a24a03a\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " Nov 03 09:45:02.757452 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757421 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-token\" (UniqueName: \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-token\") pod \"00d7493b-3271-4e42-af4b-98ac3a24a03a\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " Nov 03 09:45:02.757982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757455 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-certs\") pod \"00d7493b-3271-4e42-af4b-98ac3a24a03a\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " Nov 03 09:45:02.757982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757592 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-podinfo\") pod \"00d7493b-3271-4e42-af4b-98ac3a24a03a\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " Nov 03 09:45:02.757982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757653 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-kube-api-access-g845c\" (UniqueName: \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-kube-api-access-g845c\") pod \"00d7493b-3271-4e42-af4b-98ac3a24a03a\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " Nov 03 09:45:02.757982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757651 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-socket" pod "00d7493b-3271-4e42-af4b-98ac3a24a03a" (UID: "00d7493b-3271-4e42-af4b-98ac3a24a03a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:45:02.757982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757683 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-envoy\") pod \"00d7493b-3271-4e42-af4b-98ac3a24a03a\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " Nov 03 09:45:02.757982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757736 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-credential-socket\") pod \"00d7493b-3271-4e42-af4b-98ac3a24a03a\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " Nov 03 09:45:02.757982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757784 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-certs" pod "00d7493b-3271-4e42-af4b-98ac3a24a03a" (UID: "00d7493b-3271-4e42-af4b-98ac3a24a03a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:45:02.757982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757808 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-data\") pod \"00d7493b-3271-4e42-af4b-98ac3a24a03a\" (UID: \"00d7493b-3271-4e42-af4b-98ac3a24a03a\") " Nov 03 09:45:02.757982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.757896 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d7493b-3271-4e42-af4b-98ac3a24a03a-istiod-ca-cert" pod "00d7493b-3271-4e42-af4b-98ac3a24a03a" (UID: "00d7493b-3271-4e42-af4b-98ac3a24a03a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:45:02.758775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.758175 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-credential-socket" pod "00d7493b-3271-4e42-af4b-98ac3a24a03a" (UID: "00d7493b-3271-4e42-af4b-98ac3a24a03a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:45:02.758775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.758236 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-data" pod "00d7493b-3271-4e42-af4b-98ac3a24a03a" (UID: "00d7493b-3271-4e42-af4b-98ac3a24a03a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:45:02.758775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.758454 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:02.758775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.758519 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:02.758775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.758536 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:02.758775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.758551 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d7493b-3271-4e42-af4b-98ac3a24a03a-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:02.758775 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.758565 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:02.760162 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.760139 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-envoy" pod "00d7493b-3271-4e42-af4b-98ac3a24a03a" (UID: "00d7493b-3271-4e42-af4b-98ac3a24a03a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:45:02.760512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.760452 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-podinfo" pod "00d7493b-3271-4e42-af4b-98ac3a24a03a" (UID: "00d7493b-3271-4e42-af4b-98ac3a24a03a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:45:02.760634 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.760516 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-kube-api-access-g845c" pod "00d7493b-3271-4e42-af4b-98ac3a24a03a" (UID: "00d7493b-3271-4e42-af4b-98ac3a24a03a"). InnerVolumeSpecName "kube-api-access-g845c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:45:02.760690 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.760648 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-token" pod "00d7493b-3271-4e42-af4b-98ac3a24a03a" (UID: "00d7493b-3271-4e42-af4b-98ac3a24a03a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:45:02.859787 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.859635 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:02.859787 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.859669 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:02.859787 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.859689 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g845c\" (UniqueName: \"kubernetes.io/projected/00d7493b-3271-4e42-af4b-98ac3a24a03a-kube-api-access-g845c\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:02.859787 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:02.859704 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d7493b-3271-4e42-af4b-98ac3a24a03a-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:03.310756 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:03.310713 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m"] Nov 03 09:45:03.316271 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:03.316238 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc1444-openshift-default-c688d9cb5-kkk8m"] Nov 03 09:45:03.689601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:03.689554 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" event={"ID":"5038b440-677f-4db7-886a-79a1f6c1d965","Type":"ContainerStarted","Data":"b16a5f2c021b304f84681f04698267ca489ce166cd167fefdfdee64931226909"} Nov 03 09:45:03.692885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:03.692855 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" event={"ID":"a7642521-b6b0-4461-8dc4-da4e95811bb9","Type":"ContainerStarted","Data":"52664eac47168de9e0198be9b2b286f0ab56129ab809bc8ecc6072db64b45f9f"} Nov 03 09:45:03.694784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:03.694764 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" event={"ID":"746c99f8-0c99-4eb2-a8a0-4868089ee5c1","Type":"ContainerStarted","Data":"5b13cc8524622428d95927eca7bdeadf33cd0ac587a1f97e4f4ab8f2b2e44e65"} Nov 03 09:45:03.814160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:03.814048 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" podStartSLOduration=3.814032254 podStartE2EDuration="3.814032254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:03.813874975 +0000 UTC m=+3936.738817203" watchObservedRunningTime="2026-11-03 09:45:03.814032254 +0000 UTC m=+3936.738974527" Nov 03 09:45:04.704991 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.704953 5015 generic.go:356] "Generic (PLEG): container finished" podID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerID="52664eac47168de9e0198be9b2b286f0ab56129ab809bc8ecc6072db64b45f9f" exitCode=60 Nov 03 09:45:04.705469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.705040 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" event={"ID":"a7642521-b6b0-4461-8dc4-da4e95811bb9","Type":"ContainerDied","Data":"52664eac47168de9e0198be9b2b286f0ab56129ab809bc8ecc6072db64b45f9f"} Nov 03 09:45:04.705469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.705113 5015 scope.go:122] "RemoveContainer" containerID="8f9d171c9911c7fa34f8a93bec5c6f256846ac95732c3482eedf954676eeb355" Nov 03 09:45:04.705469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.705416 5015 scope.go:122] "RemoveContainer" containerID="52664eac47168de9e0198be9b2b286f0ab56129ab809bc8ecc6072db64b45f9f" Nov 03 09:45:04.705815 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:04.705795 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w_odh-ai-gateway-infra(a7642521-b6b0-4461-8dc4-da4e95811bb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" Nov 03 09:45:04.707348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.707315 5015 generic.go:356] "Generic (PLEG): container finished" podID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerID="5b13cc8524622428d95927eca7bdeadf33cd0ac587a1f97e4f4ab8f2b2e44e65" exitCode=60 Nov 03 09:45:04.707440 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.707397 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" event={"ID":"746c99f8-0c99-4eb2-a8a0-4868089ee5c1","Type":"ContainerDied","Data":"5b13cc8524622428d95927eca7bdeadf33cd0ac587a1f97e4f4ab8f2b2e44e65"} Nov 03 09:45:04.707644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.707622 5015 scope.go:122] "RemoveContainer" containerID="5b13cc8524622428d95927eca7bdeadf33cd0ac587a1f97e4f4ab8f2b2e44e65" Nov 03 09:45:04.707904 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:04.707885 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7_odh-ai-gateway-infra(746c99f8-0c99-4eb2-a8a0-4868089ee5c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" Nov 03 09:45:04.709277 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.709256 5015 generic.go:356] "Generic (PLEG): container finished" podID="5038b440-677f-4db7-886a-79a1f6c1d965" containerID="b16a5f2c021b304f84681f04698267ca489ce166cd167fefdfdee64931226909" exitCode=60 Nov 03 09:45:04.709367 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.709324 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" event={"ID":"5038b440-677f-4db7-886a-79a1f6c1d965","Type":"ContainerDied","Data":"b16a5f2c021b304f84681f04698267ca489ce166cd167fefdfdee64931226909"} Nov 03 09:45:04.709624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.709606 5015 scope.go:122] "RemoveContainer" containerID="b16a5f2c021b304f84681f04698267ca489ce166cd167fefdfdee64931226909" Nov 03 09:45:04.709858 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:04.709841 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9_odh-ai-gateway-infra(5038b440-677f-4db7-886a-79a1f6c1d965)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" Nov 03 09:45:04.719082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.719066 5015 scope.go:122] "RemoveContainer" containerID="8f345f2c86c15e7e98d9df07a9b5c9fde71637d8346047704911111b4b9d131d" Nov 03 09:45:04.732639 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:04.732621 5015 scope.go:122] "RemoveContainer" containerID="7683795a2631e68c5edff464f2b8f9bb24b6ae6fc5702e21693786618c344814" Nov 03 09:45:05.205845 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:05.205806 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" path="/var/lib/kubelet/pods/00d7493b-3271-4e42-af4b-98ac3a24a03a/volumes" Nov 03 09:45:05.729651 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:05.729622 5015 scope.go:122] "RemoveContainer" containerID="52664eac47168de9e0198be9b2b286f0ab56129ab809bc8ecc6072db64b45f9f" Nov 03 09:45:05.730125 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:05.729914 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w_odh-ai-gateway-infra(a7642521-b6b0-4461-8dc4-da4e95811bb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" Nov 03 09:45:05.732805 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:05.732779 5015 scope.go:122] "RemoveContainer" containerID="5b13cc8524622428d95927eca7bdeadf33cd0ac587a1f97e4f4ab8f2b2e44e65" Nov 03 09:45:05.733164 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:05.733132 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7_odh-ai-gateway-infra(746c99f8-0c99-4eb2-a8a0-4868089ee5c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" Nov 03 09:45:05.734838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:05.734818 5015 scope.go:122] "RemoveContainer" containerID="b16a5f2c021b304f84681f04698267ca489ce166cd167fefdfdee64931226909" Nov 03 09:45:05.735086 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:05.735065 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9_odh-ai-gateway-infra(5038b440-677f-4db7-886a-79a1f6c1d965)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" Nov 03 09:45:16.196898 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:16.196865 5015 scope.go:122] "RemoveContainer" containerID="52664eac47168de9e0198be9b2b286f0ab56129ab809bc8ecc6072db64b45f9f" Nov 03 09:45:16.197376 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:16.197099 5015 scope.go:122] "RemoveContainer" containerID="b16a5f2c021b304f84681f04698267ca489ce166cd167fefdfdee64931226909" Nov 03 09:45:16.826931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:16.826891 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" event={"ID":"5038b440-677f-4db7-886a-79a1f6c1d965","Type":"ContainerStarted","Data":"70ab3b75dcf94c2a5458aa03387ee458bd79173c7050ecabadea729a291950e5"} Nov 03 09:45:16.832188 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:16.832147 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" event={"ID":"a7642521-b6b0-4461-8dc4-da4e95811bb9","Type":"ContainerStarted","Data":"7715ee0fc045040fa08bbef13beaab2ca23556982a28de6394613f7600570e99"} Nov 03 09:45:16.903564 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:16.903529 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:45:17.236716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:17.236682 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z"] Nov 03 09:45:17.842693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:17.842651 5015 generic.go:356] "Generic (PLEG): container finished" podID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerID="7715ee0fc045040fa08bbef13beaab2ca23556982a28de6394613f7600570e99" exitCode=60 Nov 03 09:45:17.842907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:17.842715 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" event={"ID":"a7642521-b6b0-4461-8dc4-da4e95811bb9","Type":"ContainerDied","Data":"7715ee0fc045040fa08bbef13beaab2ca23556982a28de6394613f7600570e99"} Nov 03 09:45:17.842907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:17.842775 5015 scope.go:122] "RemoveContainer" containerID="52664eac47168de9e0198be9b2b286f0ab56129ab809bc8ecc6072db64b45f9f" Nov 03 09:45:17.843199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:17.843180 5015 scope.go:122] "RemoveContainer" containerID="7715ee0fc045040fa08bbef13beaab2ca23556982a28de6394613f7600570e99" Nov 03 09:45:17.843565 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:17.843544 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w_odh-ai-gateway-infra(a7642521-b6b0-4461-8dc4-da4e95811bb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" Nov 03 09:45:17.845859 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:17.845826 5015 generic.go:356] "Generic (PLEG): container finished" podID="5038b440-677f-4db7-886a-79a1f6c1d965" containerID="70ab3b75dcf94c2a5458aa03387ee458bd79173c7050ecabadea729a291950e5" exitCode=60 Nov 03 09:45:17.845983 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:17.845861 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" event={"ID":"5038b440-677f-4db7-886a-79a1f6c1d965","Type":"ContainerDied","Data":"70ab3b75dcf94c2a5458aa03387ee458bd79173c7050ecabadea729a291950e5"} Nov 03 09:45:17.846205 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:17.846190 5015 scope.go:122] "RemoveContainer" containerID="70ab3b75dcf94c2a5458aa03387ee458bd79173c7050ecabadea729a291950e5" Nov 03 09:45:17.846509 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:17.846467 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9_odh-ai-gateway-infra(5038b440-677f-4db7-886a-79a1f6c1d965)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" Nov 03 09:45:17.857127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:17.857099 5015 scope.go:122] "RemoveContainer" containerID="b16a5f2c021b304f84681f04698267ca489ce166cd167fefdfdee64931226909" Nov 03 09:45:17.955458 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:17.955413 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9"] Nov 03 09:45:18.075175 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:18.075140 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w"] Nov 03 09:45:18.197075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:18.196997 5015 scope.go:122] "RemoveContainer" containerID="5b13cc8524622428d95927eca7bdeadf33cd0ac587a1f97e4f4ab8f2b2e44e65" Nov 03 09:45:18.859383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:18.859348 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" event={"ID":"746c99f8-0c99-4eb2-a8a0-4868089ee5c1","Type":"ContainerStarted","Data":"dd657ca084919d63fdb8b6fe89cc01aafd87e9f82dea2dc530d4a39f4222364e"} Nov 03 09:45:19.093151 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.093117 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" Nov 03 09:45:19.130007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.128905 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" Nov 03 09:45:19.183295 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.183255 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7642521-b6b0-4461-8dc4-da4e95811bb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7642521-b6b0-4461-8dc4-da4e95811bb9-maas-api-service-ca\") pod \"a7642521-b6b0-4461-8dc4-da4e95811bb9\" (UID: \"a7642521-b6b0-4461-8dc4-da4e95811bb9\") " Nov 03 09:45:19.183504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.183342 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7642521-b6b0-4461-8dc4-da4e95811bb9-kube-api-access-htsr5\" (UniqueName: \"kubernetes.io/projected/a7642521-b6b0-4461-8dc4-da4e95811bb9-kube-api-access-htsr5\") pod \"a7642521-b6b0-4461-8dc4-da4e95811bb9\" (UID: \"a7642521-b6b0-4461-8dc4-da4e95811bb9\") " Nov 03 09:45:19.183637 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.183614 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7642521-b6b0-4461-8dc4-da4e95811bb9-maas-api-service-ca" pod "a7642521-b6b0-4461-8dc4-da4e95811bb9" (UID: "a7642521-b6b0-4461-8dc4-da4e95811bb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:45:19.184075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.184055 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7642521-b6b0-4461-8dc4-da4e95811bb9-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:19.185678 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.185652 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7642521-b6b0-4461-8dc4-da4e95811bb9-kube-api-access-htsr5" pod "a7642521-b6b0-4461-8dc4-da4e95811bb9" (UID: "a7642521-b6b0-4461-8dc4-da4e95811bb9"). InnerVolumeSpecName "kube-api-access-htsr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:45:19.285097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.285062 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5038b440-677f-4db7-886a-79a1f6c1d965-kube-api-access-5zkkk\" (UniqueName: \"kubernetes.io/projected/5038b440-677f-4db7-886a-79a1f6c1d965-kube-api-access-5zkkk\") pod \"5038b440-677f-4db7-886a-79a1f6c1d965\" (UID: \"5038b440-677f-4db7-886a-79a1f6c1d965\") " Nov 03 09:45:19.285312 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.285297 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5038b440-677f-4db7-886a-79a1f6c1d965-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5038b440-677f-4db7-886a-79a1f6c1d965-maas-api-service-ca\") pod \"5038b440-677f-4db7-886a-79a1f6c1d965\" (UID: \"5038b440-677f-4db7-886a-79a1f6c1d965\") " Nov 03 09:45:19.285793 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.285759 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5038b440-677f-4db7-886a-79a1f6c1d965-maas-api-service-ca" pod "5038b440-677f-4db7-886a-79a1f6c1d965" (UID: "5038b440-677f-4db7-886a-79a1f6c1d965"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:45:19.286953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.286929 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htsr5\" (UniqueName: \"kubernetes.io/projected/a7642521-b6b0-4461-8dc4-da4e95811bb9-kube-api-access-htsr5\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:19.287043 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.286956 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5038b440-677f-4db7-886a-79a1f6c1d965-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:19.287786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.287760 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5038b440-677f-4db7-886a-79a1f6c1d965-kube-api-access-5zkkk" pod "5038b440-677f-4db7-886a-79a1f6c1d965" (UID: "5038b440-677f-4db7-886a-79a1f6c1d965"). InnerVolumeSpecName "kube-api-access-5zkkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:45:19.388810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.388704 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zkkk\" (UniqueName: \"kubernetes.io/projected/5038b440-677f-4db7-886a-79a1f6c1d965-kube-api-access-5zkkk\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:19.878644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.878600 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" event={"ID":"5038b440-677f-4db7-886a-79a1f6c1d965","Type":"ContainerDied","Data":"83e70690d31d54ac2931d010c8f78ebd537b9a273d9c17539ee46d5f65d223f7"} Nov 03 09:45:19.879169 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.878682 5015 scope.go:122] "RemoveContainer" containerID="70ab3b75dcf94c2a5458aa03387ee458bd79173c7050ecabadea729a291950e5" Nov 03 09:45:19.879169 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.878789 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9" Nov 03 09:45:19.893855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.886142 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" Nov 03 09:45:19.893855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.886912 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w" event={"ID":"a7642521-b6b0-4461-8dc4-da4e95811bb9","Type":"ContainerDied","Data":"cea9283d8c2ddae6fd9e5221f2edb8d71a9f8326829d55704bb202b4aa6b57ce"} Nov 03 09:45:19.893855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.891202 5015 generic.go:356] "Generic (PLEG): container finished" podID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerID="dd657ca084919d63fdb8b6fe89cc01aafd87e9f82dea2dc530d4a39f4222364e" exitCode=60 Nov 03 09:45:19.893855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.891290 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" event={"ID":"746c99f8-0c99-4eb2-a8a0-4868089ee5c1","Type":"ContainerDied","Data":"dd657ca084919d63fdb8b6fe89cc01aafd87e9f82dea2dc530d4a39f4222364e"} Nov 03 09:45:19.893855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.891731 5015 scope.go:122] "RemoveContainer" containerID="dd657ca084919d63fdb8b6fe89cc01aafd87e9f82dea2dc530d4a39f4222364e" Nov 03 09:45:19.893855 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:19.892067 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7_odh-ai-gateway-infra(746c99f8-0c99-4eb2-a8a0-4868089ee5c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" Nov 03 09:45:19.903699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.903672 5015 scope.go:122] "RemoveContainer" containerID="7715ee0fc045040fa08bbef13beaab2ca23556982a28de6394613f7600570e99" Nov 03 09:45:19.915057 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:19.915034 5015 scope.go:122] "RemoveContainer" containerID="5b13cc8524622428d95927eca7bdeadf33cd0ac587a1f97e4f4ab8f2b2e44e65" Nov 03 09:45:20.144401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:20.144361 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7"] Nov 03 09:45:20.171760 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:20.171724 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w"] Nov 03 09:45:20.174869 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:20.174847 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894985-5sn4w"] Nov 03 09:45:20.291403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:20.291373 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9"] Nov 03 09:45:20.297286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:20.297259 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894985-nq5n9"] Nov 03 09:45:21.078357 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.078327 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" Nov 03 09:45:21.208635 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.208553 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" path="/var/lib/kubelet/pods/5038b440-677f-4db7-886a-79a1f6c1d965/volumes" Nov 03 09:45:21.209056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.209038 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" path="/var/lib/kubelet/pods/a7642521-b6b0-4461-8dc4-da4e95811bb9/volumes" Nov 03 09:45:21.211205 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.211179 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-kube-api-access-thc4s\" (UniqueName: \"kubernetes.io/projected/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-kube-api-access-thc4s\") pod \"746c99f8-0c99-4eb2-a8a0-4868089ee5c1\" (UID: \"746c99f8-0c99-4eb2-a8a0-4868089ee5c1\") " Nov 03 09:45:21.211342 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.211283 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-maas-api-service-ca\") pod \"746c99f8-0c99-4eb2-a8a0-4868089ee5c1\" (UID: \"746c99f8-0c99-4eb2-a8a0-4868089ee5c1\") " Nov 03 09:45:21.211889 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.211860 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-maas-api-service-ca" pod "746c99f8-0c99-4eb2-a8a0-4868089ee5c1" (UID: "746c99f8-0c99-4eb2-a8a0-4868089ee5c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:45:21.212124 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.212106 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:21.213909 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.213886 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-kube-api-access-thc4s" pod "746c99f8-0c99-4eb2-a8a0-4868089ee5c1" (UID: "746c99f8-0c99-4eb2-a8a0-4868089ee5c1"). InnerVolumeSpecName "kube-api-access-thc4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:45:21.313944 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.313909 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thc4s\" (UniqueName: \"kubernetes.io/projected/746c99f8-0c99-4eb2-a8a0-4868089ee5c1-kube-api-access-thc4s\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:21.917570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.917540 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" Nov 03 09:45:21.917759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.917539 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7" event={"ID":"746c99f8-0c99-4eb2-a8a0-4868089ee5c1","Type":"ContainerDied","Data":"663d9de9dfac7f41dd70a0eef099a753d416d58e1f6c3da124c281423832f9a3"} Nov 03 09:45:21.917759 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:21.917670 5015 scope.go:122] "RemoveContainer" containerID="dd657ca084919d63fdb8b6fe89cc01aafd87e9f82dea2dc530d4a39f4222364e" Nov 03 09:45:22.087003 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.086971 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7"] Nov 03 09:45:22.139567 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.139534 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894985-vqfm7"] Nov 03 09:45:22.238546 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.238510 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" podUID="46818ade-3cc4-42d9-8ab2-b0795d09213a" containerName="payload-pre-processing" containerID="cri-o://1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4" gracePeriod=25 Nov 03 09:45:22.515218 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.515190 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:45:22.631281 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.631251 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46818ade-3cc4-42d9-8ab2-b0795d09213a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46818ade-3cc4-42d9-8ab2-b0795d09213a-plugins-config-volume\") pod \"46818ade-3cc4-42d9-8ab2-b0795d09213a\" (UID: \"46818ade-3cc4-42d9-8ab2-b0795d09213a\") " Nov 03 09:45:22.631524 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.631361 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46818ade-3cc4-42d9-8ab2-b0795d09213a-kube-api-access-8hk4h\" (UniqueName: \"kubernetes.io/projected/46818ade-3cc4-42d9-8ab2-b0795d09213a-kube-api-access-8hk4h\") pod \"46818ade-3cc4-42d9-8ab2-b0795d09213a\" (UID: \"46818ade-3cc4-42d9-8ab2-b0795d09213a\") " Nov 03 09:45:22.632280 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.632249 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46818ade-3cc4-42d9-8ab2-b0795d09213a-plugins-config-volume" pod "46818ade-3cc4-42d9-8ab2-b0795d09213a" (UID: "46818ade-3cc4-42d9-8ab2-b0795d09213a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:45:22.634779 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.634741 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46818ade-3cc4-42d9-8ab2-b0795d09213a-kube-api-access-8hk4h" pod "46818ade-3cc4-42d9-8ab2-b0795d09213a" (UID: "46818ade-3cc4-42d9-8ab2-b0795d09213a"). InnerVolumeSpecName "kube-api-access-8hk4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:45:22.733060 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.733024 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46818ade-3cc4-42d9-8ab2-b0795d09213a-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:22.733060 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.733053 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hk4h\" (UniqueName: \"kubernetes.io/projected/46818ade-3cc4-42d9-8ab2-b0795d09213a-kube-api-access-8hk4h\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:22.926812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.926716 5015 generic.go:356] "Generic (PLEG): container finished" podID="46818ade-3cc4-42d9-8ab2-b0795d09213a" containerID="1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4" exitCode=0 Nov 03 09:45:22.926812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.926758 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" event={"ID":"46818ade-3cc4-42d9-8ab2-b0795d09213a","Type":"ContainerDied","Data":"1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4"} Nov 03 09:45:22.926812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.926765 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" Nov 03 09:45:22.926812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.926797 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z" event={"ID":"46818ade-3cc4-42d9-8ab2-b0795d09213a","Type":"ContainerDied","Data":"ba96e0485693b98704506a80c69549123336e260ba3e6094d0b3a19eb2f205b3"} Nov 03 09:45:22.927131 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.926820 5015 scope.go:122] "RemoveContainer" containerID="1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4" Nov 03 09:45:22.938453 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.938426 5015 scope.go:122] "RemoveContainer" containerID="1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4" Nov 03 09:45:22.938758 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:22.938726 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4\": container with ID starting with 1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4 not found: ID does not exist" containerID="1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4" Nov 03 09:45:22.938819 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:22.938758 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4"} err="failed to get container status \"1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4\": rpc error: code = NotFound desc = could not find container \"1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4\": container with ID starting with 1651191a5a8219b300802b23dc08c716cbea82d25589e5fad274559909eafdf4 not found: ID does not exist" Nov 03 09:45:23.096663 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:23.096633 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z"] Nov 03 09:45:23.104359 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:23.104326 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af514d-b98769899-7nx7z"] Nov 03 09:45:23.206699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:23.206622 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46818ade-3cc4-42d9-8ab2-b0795d09213a" path="/var/lib/kubelet/pods/46818ade-3cc4-42d9-8ab2-b0795d09213a/volumes" Nov 03 09:45:23.207080 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:45:23.207066 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" path="/var/lib/kubelet/pods/746c99f8-0c99-4eb2-a8a0-4868089ee5c1/volumes" Nov 03 09:45:46.904611 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:45:46.904571 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:46:16.904504 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:46:16.904382 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:46:24.565646 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.565609 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz"] Nov 03 09:46:24.566084 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.565818 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" podUID="dc96794c-ba60-4823-8aac-9bbd472cff0e" containerName="maas-api" containerID="cri-o://36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e" gracePeriod=30 Nov 03 09:46:24.860334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.860307 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:46:24.965994 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.965953 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-metrics-certs\") pod \"dc96794c-ba60-4823-8aac-9bbd472cff0e\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " Nov 03 09:46:24.966229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.966092 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-maas-api-tls\") pod \"dc96794c-ba60-4823-8aac-9bbd472cff0e\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " Nov 03 09:46:24.966229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.966189 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc96794c-ba60-4823-8aac-9bbd472cff0e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc96794c-ba60-4823-8aac-9bbd472cff0e-openshift-service-ca\") pod \"dc96794c-ba60-4823-8aac-9bbd472cff0e\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " Nov 03 09:46:24.966229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.966224 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc96794c-ba60-4823-8aac-9bbd472cff0e-kube-api-access-vsccv\" (UniqueName: \"kubernetes.io/projected/dc96794c-ba60-4823-8aac-9bbd472cff0e-kube-api-access-vsccv\") pod \"dc96794c-ba60-4823-8aac-9bbd472cff0e\" (UID: \"dc96794c-ba60-4823-8aac-9bbd472cff0e\") " Nov 03 09:46:24.967045 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.966565 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc96794c-ba60-4823-8aac-9bbd472cff0e-openshift-service-ca" pod "dc96794c-ba60-4823-8aac-9bbd472cff0e" (UID: "dc96794c-ba60-4823-8aac-9bbd472cff0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:46:24.967045 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.966781 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc96794c-ba60-4823-8aac-9bbd472cff0e-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:24.968622 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.968588 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-maas-api-tls" pod "dc96794c-ba60-4823-8aac-9bbd472cff0e" (UID: "dc96794c-ba60-4823-8aac-9bbd472cff0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:46:24.968745 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.968625 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-metrics-certs" pod "dc96794c-ba60-4823-8aac-9bbd472cff0e" (UID: "dc96794c-ba60-4823-8aac-9bbd472cff0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:46:24.969662 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:24.969636 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc96794c-ba60-4823-8aac-9bbd472cff0e-kube-api-access-vsccv" pod "dc96794c-ba60-4823-8aac-9bbd472cff0e" (UID: "dc96794c-ba60-4823-8aac-9bbd472cff0e"). InnerVolumeSpecName "kube-api-access-vsccv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:46:25.068389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.068295 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:25.068389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.068334 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc96794c-ba60-4823-8aac-9bbd472cff0e-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:25.068389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.068350 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsccv\" (UniqueName: \"kubernetes.io/projected/dc96794c-ba60-4823-8aac-9bbd472cff0e-kube-api-access-vsccv\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:25.492621 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.492581 5015 generic.go:356] "Generic (PLEG): container finished" podID="dc96794c-ba60-4823-8aac-9bbd472cff0e" containerID="36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e" exitCode=0 Nov 03 09:46:25.492811 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.492634 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" Nov 03 09:46:25.492811 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.492659 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" event={"ID":"dc96794c-ba60-4823-8aac-9bbd472cff0e","Type":"ContainerDied","Data":"36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e"} Nov 03 09:46:25.492811 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.492716 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz" event={"ID":"dc96794c-ba60-4823-8aac-9bbd472cff0e","Type":"ContainerDied","Data":"efed1c606db149dacb3fdb0d6c124886836c19ce1af1e5b7fc13700aca3f710b"} Nov 03 09:46:25.492811 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.492736 5015 scope.go:122] "RemoveContainer" containerID="36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e" Nov 03 09:46:25.503390 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.503371 5015 scope.go:122] "RemoveContainer" containerID="36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e" Nov 03 09:46:25.503770 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:46:25.503748 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e\": container with ID starting with 36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e not found: ID does not exist" containerID="36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e" Nov 03 09:46:25.503865 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.503779 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e"} err="failed to get container status \"36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e\": rpc error: code = NotFound desc = could not find container \"36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e\": container with ID starting with 36c9e8362095687493a28dbce54662fbd2f32b77cd9eb4b5fefc706c23671a1e not found: ID does not exist" Nov 03 09:46:25.682147 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.682114 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz"] Nov 03 09:46:25.730531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:25.730502 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af514d-6f44bb85cc-m2vnz"] Nov 03 09:46:27.206736 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:46:27.206701 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc96794c-ba60-4823-8aac-9bbd472cff0e" path="/var/lib/kubelet/pods/dc96794c-ba60-4823-8aac-9bbd472cff0e/volumes" Nov 03 09:46:46.902717 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:46:46.902671 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:47:16.903730 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:47:16.903684 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:47:46.903557 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:47:46.903450 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:48:16.903431 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:48:16.903387 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:48:39.876446 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.876405 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn"] Nov 03 09:48:39.877336 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877318 5015 cpu_manager.go:414] "containerMap: removing container" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerName="cleanup" Nov 03 09:48:39.877382 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877339 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerName="cleanup" Nov 03 09:48:39.877382 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877355 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerName="cleanup" Nov 03 09:48:39.877382 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877363 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerName="cleanup" Nov 03 09:48:39.877509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877382 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerName="cleanup" Nov 03 09:48:39.877509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877392 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerName="cleanup" Nov 03 09:48:39.877509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877421 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerName="cleanup" Nov 03 09:48:39.877509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877429 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerName="cleanup" Nov 03 09:48:39.877509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877446 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" containerName="cleanup" Nov 03 09:48:39.877509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877454 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" containerName="cleanup" Nov 03 09:48:39.877509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877467 5015 cpu_manager.go:414] "containerMap: removing container" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" Nov 03 09:48:39.877509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877496 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" Nov 03 09:48:39.877509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877507 5015 cpu_manager.go:414] "containerMap: removing container" podUID="46818ade-3cc4-42d9-8ab2-b0795d09213a" containerName="payload-pre-processing" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877515 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46818ade-3cc4-42d9-8ab2-b0795d09213a" containerName="payload-pre-processing" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877534 5015 cpu_manager.go:414] "containerMap: removing container" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerName="cleanup" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877540 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerName="cleanup" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877553 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" containerName="cleanup" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877560 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" containerName="cleanup" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877573 5015 cpu_manager.go:414] "containerMap: removing container" podUID="dc96794c-ba60-4823-8aac-9bbd472cff0e" containerName="maas-api" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877579 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc96794c-ba60-4823-8aac-9bbd472cff0e" containerName="maas-api" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877595 5015 cpu_manager.go:414] "containerMap: removing container" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerName="cleanup" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877602 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerName="cleanup" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877757 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerName="cleanup" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877769 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" containerName="cleanup" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877778 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877787 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="46818ade-3cc4-42d9-8ab2-b0795d09213a" containerName="payload-pre-processing" Nov 03 09:48:39.877794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877795 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" containerName="cleanup" Nov 03 09:48:39.878208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877806 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerName="cleanup" Nov 03 09:48:39.878208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877814 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" containerName="cleanup" Nov 03 09:48:39.878208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877822 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc96794c-ba60-4823-8aac-9bbd472cff0e" containerName="maas-api" Nov 03 09:48:39.878208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877833 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerName="cleanup" Nov 03 09:48:39.878208 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.877844 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerName="cleanup" Nov 03 09:48:39.882270 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.882249 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:39.944637 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.944601 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn"] Nov 03 09:48:39.988715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.988676 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-433821\"/\"default-dockercfg-mvdq5\"" Nov 03 09:48:39.988715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.988692 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-433821\"/\"rate-test-model-433821-kserve-self-signed-certs\"" Nov 03 09:48:39.988715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.988692 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-433821\"/\"openshift-service-ca.crt\"" Nov 03 09:48:39.988989 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.988761 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-433821\"/\"kube-root-ca.crt\"" Nov 03 09:48:39.997352 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.997325 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/384591ef-c634-4c99-8e9b-4f85332e5802-tls-certs\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:39.997467 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.997381 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-dshm\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:39.997572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.997521 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-tmp-dir\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:39.997611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.997579 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-model-cache\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:39.997735 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.997716 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbp95\" (UniqueName: \"kubernetes.io/projected/384591ef-c634-4c99-8e9b-4f85332e5802-kube-api-access-hbp95\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:39.997773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:39.997756 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-home\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.099213 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.099183 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbp95\" (UniqueName: \"kubernetes.io/projected/384591ef-c634-4c99-8e9b-4f85332e5802-kube-api-access-hbp95\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.099395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.099227 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-home\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.099442 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.099404 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/384591ef-c634-4c99-8e9b-4f85332e5802-tls-certs\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.099519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.099496 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-dshm\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.099586 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.099572 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-tmp-dir\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.099647 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.099629 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-model-cache\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.099708 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.099651 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-home\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.099928 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.099905 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-tmp-dir\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.100005 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.099922 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-model-cache\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.101962 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.101940 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-dshm\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.102248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.102229 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/384591ef-c634-4c99-8e9b-4f85332e5802-tls-certs\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.159008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.158941 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbp95\" (UniqueName: \"kubernetes.io/projected/384591ef-c634-4c99-8e9b-4f85332e5802-kube-api-access-hbp95\") pod \"rate-test-model-433821-kserve-5798c67fc7-bfmqn\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.197955 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.197923 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:40.582972 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.582938 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn"] Nov 03 09:48:40.583205 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:48:40.583178 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod384591ef_c634_4c99_8e9b_4f85332e5802.slice/crio-e944680373b4aeb6cfb0fdc23edb6e0f198f352547c51aa77e0536fb2b5e3ee2 WatchSource:0}: Error finding container e944680373b4aeb6cfb0fdc23edb6e0f198f352547c51aa77e0536fb2b5e3ee2: Status 404 returned error can't find the container with id e944680373b4aeb6cfb0fdc23edb6e0f198f352547c51aa77e0536fb2b5e3ee2 Nov 03 09:48:40.617077 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:40.617042 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" event={"ID":"384591ef-c634-4c99-8e9b-4f85332e5802","Type":"ContainerStarted","Data":"e944680373b4aeb6cfb0fdc23edb6e0f198f352547c51aa77e0536fb2b5e3ee2"} Nov 03 09:48:41.626062 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:41.626023 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" event={"ID":"384591ef-c634-4c99-8e9b-4f85332e5802","Type":"ContainerStarted","Data":"ccfcf9681929097698e858fa30edc139bd4915e85b46e26a5c56e29b2d02e6f0"} Nov 03 09:48:41.742950 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:41.742893 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" podStartSLOduration=2.742877973 podStartE2EDuration="2.742877973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:48:41.741251032 +0000 UTC m=+4154.666193260" watchObservedRunningTime="2026-11-03 09:48:41.742877973 +0000 UTC m=+4154.667820191" Nov 03 09:48:42.198299 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:42.198266 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:46.902989 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:48:46.902952 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:48:50.199097 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:50.199052 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:50.212111 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:50.212083 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:50.715117 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:50.715078 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:48:53.025907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.025876 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5"] Nov 03 09:48:53.026574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.026557 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" containerName="cleanup" Nov 03 09:48:53.026574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.026575 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5038b440-677f-4db7-886a-79a1f6c1d965" containerName="cleanup" Nov 03 09:48:53.026695 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.026624 5015 cpu_manager.go:414] "containerMap: removing container" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" Nov 03 09:48:53.026695 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.026631 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" Nov 03 09:48:53.026765 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.026723 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7642521-b6b0-4461-8dc4-da4e95811bb9" containerName="cleanup" Nov 03 09:48:53.026765 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.026734 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d7493b-3271-4e42-af4b-98ac3a24a03a" containerName="istio-proxy" Nov 03 09:48:53.026765 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.026744 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="746c99f8-0c99-4eb2-a8a0-4868089ee5c1" containerName="cleanup" Nov 03 09:48:53.030395 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.030375 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.068128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.068089 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-home\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.068324 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.068292 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783d720f-d297-495b-b96a-299116fd1150-tls-certs\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.068450 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.068431 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-dshm\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.068707 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.068686 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-model-cache\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.069075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.068716 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-tmp-dir\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.069075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.068757 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7tg5\" (UniqueName: \"kubernetes.io/projected/783d720f-d297-495b-b96a-299116fd1150-kube-api-access-g7tg5\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.088584 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.088437 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5"] Nov 03 09:48:53.132565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.132526 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-f9b517\"/\"default-dockercfg-2s2sn\"" Nov 03 09:48:53.132717 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.132578 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-f9b517\"/\"openshift-service-ca.crt\"" Nov 03 09:48:53.132717 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.132619 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-f9b517\"/\"rate-test-model-f9b517-kserve-self-signed-certs\"" Nov 03 09:48:53.134042 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.134020 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-f9b517\"/\"kube-root-ca.crt\"" Nov 03 09:48:53.170327 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.170294 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-dshm\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.170516 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.170364 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-model-cache\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.170516 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.170392 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-tmp-dir\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.170516 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.170425 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7tg5\" (UniqueName: \"kubernetes.io/projected/783d720f-d297-495b-b96a-299116fd1150-kube-api-access-g7tg5\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.170516 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.170512 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-home\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.170757 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.170645 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783d720f-d297-495b-b96a-299116fd1150-tls-certs\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.170880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.170854 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-tmp-dir\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.170946 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.170903 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-home\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.171004 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.170974 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-model-cache\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.172860 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.172834 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-dshm\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.173421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.173398 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783d720f-d297-495b-b96a-299116fd1150-tls-certs\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.241222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.241183 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7tg5\" (UniqueName: \"kubernetes.io/projected/783d720f-d297-495b-b96a-299116fd1150-kube-api-access-g7tg5\") pod \"rate-test-model-f9b517-kserve-6dddf5cc49-szls5\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.344095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.344022 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:53.729362 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:48:53.729334 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod783d720f_d297_495b_b96a_299116fd1150.slice/crio-5b8fe508702ece94d17097d0b19751260a5026ac4a2ba41928e1f07989511fea WatchSource:0}: Error finding container 5b8fe508702ece94d17097d0b19751260a5026ac4a2ba41928e1f07989511fea: Status 404 returned error can't find the container with id 5b8fe508702ece94d17097d0b19751260a5026ac4a2ba41928e1f07989511fea Nov 03 09:48:53.729754 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:53.729728 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5"] Nov 03 09:48:54.743077 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:54.743022 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" event={"ID":"783d720f-d297-495b-b96a-299116fd1150","Type":"ContainerStarted","Data":"05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc"} Nov 03 09:48:54.743624 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:54.743093 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" event={"ID":"783d720f-d297-495b-b96a-299116fd1150","Type":"ContainerStarted","Data":"5b8fe508702ece94d17097d0b19751260a5026ac4a2ba41928e1f07989511fea"} Nov 03 09:48:54.860911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:54.860851 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" podStartSLOduration=2.860834425 podStartE2EDuration="2.860834425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:48:54.858877137 +0000 UTC m=+4167.783819365" watchObservedRunningTime="2026-11-03 09:48:54.860834425 +0000 UTC m=+4167.785776683" Nov 03 09:48:55.345039 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:55.345006 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:48:57.849963 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:57.849923 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c"] Nov 03 09:48:57.960599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:48:57.960551 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6"] Nov 03 09:49:02.851180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:02.851139 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" podUID="6bf84a87-47f3-4c80-a429-564a523574e6" containerName="payload-processing" containerID="cri-o://c8d5f679592cf8ba3db7ec1a718bf50585fe35a20b2dd1761a47111043148ebb" gracePeriod=25 Nov 03 09:49:02.961761 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:02.961683 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" podUID="2ec048a3-b147-4650-8949-25377a53aa41" containerName="payload-pre-processing" containerID="cri-o://53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8" gracePeriod=25 Nov 03 09:49:03.344559 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:03.344524 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:49:03.358106 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:03.358080 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:49:03.831361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:03.831332 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:49:08.255831 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:08.255749 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" podUID="6bf84a87-47f3-4c80-a429-564a523574e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:49:08.255831 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:08.255745 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" podUID="2ec048a3-b147-4650-8949-25377a53aa41" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:49:16.902533 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:49:16.902457 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:49:18.255045 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:18.254998 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" podUID="6bf84a87-47f3-4c80-a429-564a523574e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:49:18.255593 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:18.255016 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" podUID="2ec048a3-b147-4650-8949-25377a53aa41" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:49:23.002570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.002536 5015 generic.go:356] "Generic (PLEG): container finished" podID="6bf84a87-47f3-4c80-a429-564a523574e6" containerID="c8d5f679592cf8ba3db7ec1a718bf50585fe35a20b2dd1761a47111043148ebb" exitCode=0 Nov 03 09:49:23.002962 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.002619 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" event={"ID":"6bf84a87-47f3-4c80-a429-564a523574e6","Type":"ContainerDied","Data":"c8d5f679592cf8ba3db7ec1a718bf50585fe35a20b2dd1761a47111043148ebb"} Nov 03 09:49:23.150954 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.150919 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:49:23.250361 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.250325 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf84a87-47f3-4c80-a429-564a523574e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf84a87-47f3-4c80-a429-564a523574e6-plugins-config-volume\") pod \"6bf84a87-47f3-4c80-a429-564a523574e6\" (UID: \"6bf84a87-47f3-4c80-a429-564a523574e6\") " Nov 03 09:49:23.250567 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.250396 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf84a87-47f3-4c80-a429-564a523574e6-kube-api-access-jljkk\" (UniqueName: \"kubernetes.io/projected/6bf84a87-47f3-4c80-a429-564a523574e6-kube-api-access-jljkk\") pod \"6bf84a87-47f3-4c80-a429-564a523574e6\" (UID: \"6bf84a87-47f3-4c80-a429-564a523574e6\") " Nov 03 09:49:23.250942 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.250911 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf84a87-47f3-4c80-a429-564a523574e6-plugins-config-volume" pod "6bf84a87-47f3-4c80-a429-564a523574e6" (UID: "6bf84a87-47f3-4c80-a429-564a523574e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:49:23.251166 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.251148 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf84a87-47f3-4c80-a429-564a523574e6-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:23.254750 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.254718 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf84a87-47f3-4c80-a429-564a523574e6-kube-api-access-jljkk" pod "6bf84a87-47f3-4c80-a429-564a523574e6" (UID: "6bf84a87-47f3-4c80-a429-564a523574e6"). InnerVolumeSpecName "kube-api-access-jljkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:49:23.265862 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.265828 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:49:23.352728 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.352696 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec048a3-b147-4650-8949-25377a53aa41-kube-api-access-vspc5\" (UniqueName: \"kubernetes.io/projected/2ec048a3-b147-4650-8949-25377a53aa41-kube-api-access-vspc5\") pod \"2ec048a3-b147-4650-8949-25377a53aa41\" (UID: \"2ec048a3-b147-4650-8949-25377a53aa41\") " Nov 03 09:49:23.352942 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.352801 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec048a3-b147-4650-8949-25377a53aa41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec048a3-b147-4650-8949-25377a53aa41-plugins-config-volume\") pod \"2ec048a3-b147-4650-8949-25377a53aa41\" (UID: \"2ec048a3-b147-4650-8949-25377a53aa41\") " Nov 03 09:49:23.353218 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.353188 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec048a3-b147-4650-8949-25377a53aa41-plugins-config-volume" pod "2ec048a3-b147-4650-8949-25377a53aa41" (UID: "2ec048a3-b147-4650-8949-25377a53aa41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:49:23.353460 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.353435 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jljkk\" (UniqueName: \"kubernetes.io/projected/6bf84a87-47f3-4c80-a429-564a523574e6-kube-api-access-jljkk\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:23.353635 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.353559 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec048a3-b147-4650-8949-25377a53aa41-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:23.355275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.355215 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec048a3-b147-4650-8949-25377a53aa41-kube-api-access-vspc5" pod "2ec048a3-b147-4650-8949-25377a53aa41" (UID: "2ec048a3-b147-4650-8949-25377a53aa41"). InnerVolumeSpecName "kube-api-access-vspc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:49:23.455144 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:23.455067 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vspc5\" (UniqueName: \"kubernetes.io/projected/2ec048a3-b147-4650-8949-25377a53aa41-kube-api-access-vspc5\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:24.011507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.011455 5015 generic.go:356] "Generic (PLEG): container finished" podID="2ec048a3-b147-4650-8949-25377a53aa41" containerID="53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8" exitCode=0 Nov 03 09:49:24.011929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.011513 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" event={"ID":"2ec048a3-b147-4650-8949-25377a53aa41","Type":"ContainerDied","Data":"53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8"} Nov 03 09:49:24.011929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.011533 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" Nov 03 09:49:24.011929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.011580 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6" event={"ID":"2ec048a3-b147-4650-8949-25377a53aa41","Type":"ContainerDied","Data":"e1191da052364757a8af13c046ea220807250ca407c4b6af301e3afff34c6a39"} Nov 03 09:49:24.011929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.011611 5015 scope.go:122] "RemoveContainer" containerID="53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8" Nov 03 09:49:24.013796 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.013780 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" Nov 03 09:49:24.013796 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.013789 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c" event={"ID":"6bf84a87-47f3-4c80-a429-564a523574e6","Type":"ContainerDied","Data":"ce6d88a9a633654d0b9d9977ff765d8a4708cad94717a76ea380622c54df2617"} Nov 03 09:49:24.024729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.024711 5015 scope.go:122] "RemoveContainer" containerID="53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8" Nov 03 09:49:24.025039 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:49:24.025011 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8\": container with ID starting with 53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8 not found: ID does not exist" containerID="53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8" Nov 03 09:49:24.025160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.025050 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8"} err="failed to get container status \"53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8\": rpc error: code = NotFound desc = could not find container \"53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8\": container with ID starting with 53b21798c4e62ae997e05882b34cadd78db545a83e83a599ddb2697c52d3e0a8 not found: ID does not exist" Nov 03 09:49:24.025160 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.025074 5015 scope.go:122] "RemoveContainer" containerID="c8d5f679592cf8ba3db7ec1a718bf50585fe35a20b2dd1761a47111043148ebb" Nov 03 09:49:24.181871 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.181790 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6"] Nov 03 09:49:24.231947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.231916 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-465707-78cdd85cd7-jkrn6"] Nov 03 09:49:24.299957 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.299923 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c"] Nov 03 09:49:24.303945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:24.303922 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-465707-7957dd57fc-f5s4c"] Nov 03 09:49:25.208432 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:25.208397 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec048a3-b147-4650-8949-25377a53aa41" path="/var/lib/kubelet/pods/2ec048a3-b147-4650-8949-25377a53aa41/volumes" Nov 03 09:49:25.209012 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:25.208982 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf84a87-47f3-4c80-a429-564a523574e6" path="/var/lib/kubelet/pods/6bf84a87-47f3-4c80-a429-564a523574e6/volumes" Nov 03 09:49:26.705201 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:26.705167 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk"] Nov 03 09:49:26.705692 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:26.705419 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" podUID="e7b5cc05-6c87-427d-8691-8bfe297128a7" containerName="maas-api" containerID="cri-o://5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639" gracePeriod=30 Nov 03 09:49:27.001104 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.001080 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:49:27.040873 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.040840 5015 generic.go:356] "Generic (PLEG): container finished" podID="e7b5cc05-6c87-427d-8691-8bfe297128a7" containerID="5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639" exitCode=0 Nov 03 09:49:27.041055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.040893 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" Nov 03 09:49:27.041055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.040934 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" event={"ID":"e7b5cc05-6c87-427d-8691-8bfe297128a7","Type":"ContainerDied","Data":"5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639"} Nov 03 09:49:27.041055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.040989 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk" event={"ID":"e7b5cc05-6c87-427d-8691-8bfe297128a7","Type":"ContainerDied","Data":"6a8c1e6d2240f66f53d628680a33aadb01fefcb6f0f295341f0e73b9241d21b6"} Nov 03 09:49:27.041055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.041013 5015 scope.go:122] "RemoveContainer" containerID="5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639" Nov 03 09:49:27.051967 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.051950 5015 scope.go:122] "RemoveContainer" containerID="5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639" Nov 03 09:49:27.052235 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:49:27.052216 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639\": container with ID starting with 5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639 not found: ID does not exist" containerID="5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639" Nov 03 09:49:27.052286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.052248 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639"} err="failed to get container status \"5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639\": rpc error: code = NotFound desc = could not find container \"5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639\": container with ID starting with 5d958baa07cb7b7eb5fd1df0598960e00be454bc2cd07e74000a0715728ef639 not found: ID does not exist" Nov 03 09:49:27.101880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.101846 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-maas-api-tls\") pod \"e7b5cc05-6c87-427d-8691-8bfe297128a7\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " Nov 03 09:49:27.102036 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.101917 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-metrics-certs\") pod \"e7b5cc05-6c87-427d-8691-8bfe297128a7\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " Nov 03 09:49:27.102036 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.101987 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7b5cc05-6c87-427d-8691-8bfe297128a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b5cc05-6c87-427d-8691-8bfe297128a7-openshift-service-ca\") pod \"e7b5cc05-6c87-427d-8691-8bfe297128a7\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " Nov 03 09:49:27.102036 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.102011 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b5cc05-6c87-427d-8691-8bfe297128a7-kube-api-access-8rm8q\" (UniqueName: \"kubernetes.io/projected/e7b5cc05-6c87-427d-8691-8bfe297128a7-kube-api-access-8rm8q\") pod \"e7b5cc05-6c87-427d-8691-8bfe297128a7\" (UID: \"e7b5cc05-6c87-427d-8691-8bfe297128a7\") " Nov 03 09:49:27.102378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.102351 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b5cc05-6c87-427d-8691-8bfe297128a7-openshift-service-ca" pod "e7b5cc05-6c87-427d-8691-8bfe297128a7" (UID: "e7b5cc05-6c87-427d-8691-8bfe297128a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:49:27.102543 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.102527 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b5cc05-6c87-427d-8691-8bfe297128a7-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:27.104468 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.104354 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-metrics-certs" pod "e7b5cc05-6c87-427d-8691-8bfe297128a7" (UID: "e7b5cc05-6c87-427d-8691-8bfe297128a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:49:27.104468 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.104405 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-maas-api-tls" pod "e7b5cc05-6c87-427d-8691-8bfe297128a7" (UID: "e7b5cc05-6c87-427d-8691-8bfe297128a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:49:27.104468 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.104415 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b5cc05-6c87-427d-8691-8bfe297128a7-kube-api-access-8rm8q" pod "e7b5cc05-6c87-427d-8691-8bfe297128a7" (UID: "e7b5cc05-6c87-427d-8691-8bfe297128a7"). InnerVolumeSpecName "kube-api-access-8rm8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:49:27.203923 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.203888 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rm8q\" (UniqueName: \"kubernetes.io/projected/e7b5cc05-6c87-427d-8691-8bfe297128a7-kube-api-access-8rm8q\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:27.203923 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.203916 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:27.203923 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.203926 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b5cc05-6c87-427d-8691-8bfe297128a7-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:27.467140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.467099 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk"] Nov 03 09:49:27.467362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:27.467157 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-465707-7f589fc5b9-9mlmk"] Nov 03 09:49:28.008063 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:28.008032 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:49:28.031405 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:28.031373 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:49:29.207722 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:29.207691 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7b5cc05-6c87-427d-8691-8bfe297128a7" path="/var/lib/kubelet/pods/e7b5cc05-6c87-427d-8691-8bfe297128a7/volumes" Nov 03 09:49:41.736520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:41.736461 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs"] Nov 03 09:49:46.238238 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.238203 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv"] Nov 03 09:49:46.239035 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.239013 5015 cpu_manager.go:414] "containerMap: removing container" podUID="e7b5cc05-6c87-427d-8691-8bfe297128a7" containerName="maas-api" Nov 03 09:49:46.239144 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.239037 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7b5cc05-6c87-427d-8691-8bfe297128a7" containerName="maas-api" Nov 03 09:49:46.239144 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.239083 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6bf84a87-47f3-4c80-a429-564a523574e6" containerName="payload-processing" Nov 03 09:49:46.239144 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.239092 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf84a87-47f3-4c80-a429-564a523574e6" containerName="payload-processing" Nov 03 09:49:46.239144 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.239116 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2ec048a3-b147-4650-8949-25377a53aa41" containerName="payload-pre-processing" Nov 03 09:49:46.239144 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.239124 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec048a3-b147-4650-8949-25377a53aa41" containerName="payload-pre-processing" Nov 03 09:49:46.239413 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.239332 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7b5cc05-6c87-427d-8691-8bfe297128a7" containerName="maas-api" Nov 03 09:49:46.239413 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.239353 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf84a87-47f3-4c80-a429-564a523574e6" containerName="payload-processing" Nov 03 09:49:46.239413 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.239366 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec048a3-b147-4650-8949-25377a53aa41" containerName="payload-pre-processing" Nov 03 09:49:46.243248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.243228 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.300605 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.300562 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv"] Nov 03 09:49:46.346284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.346254 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-506e6d\"/\"rate-test-model-506e6d-kserve-self-signed-certs\"" Nov 03 09:49:46.346505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.346259 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-506e6d\"/\"default-dockercfg-s654v\"" Nov 03 09:49:46.346505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.346303 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-506e6d\"/\"kube-root-ca.crt\"" Nov 03 09:49:46.347004 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.346982 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-506e6d\"/\"openshift-service-ca.crt\"" Nov 03 09:49:46.349726 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.349702 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-model-cache\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.349872 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.349846 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r92m\" (UniqueName: \"kubernetes.io/projected/373ca1dd-aa36-4d69-af70-2970f27c1c97-kube-api-access-6r92m\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.350090 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.350015 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-home\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.350090 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.350071 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373ca1dd-aa36-4d69-af70-2970f27c1c97-tls-certs\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.350272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.350137 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-dshm\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.350272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.350181 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-tmp-dir\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.451492 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.451449 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-home\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.451683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.451541 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373ca1dd-aa36-4d69-af70-2970f27c1c97-tls-certs\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.451683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.451600 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-dshm\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.451683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.451634 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-tmp-dir\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.451930 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.451902 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-home\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.452079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.451951 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-tmp-dir\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.452079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.452060 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-model-cache\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.452198 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.452161 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r92m\" (UniqueName: \"kubernetes.io/projected/373ca1dd-aa36-4d69-af70-2970f27c1c97-kube-api-access-6r92m\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.452570 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.452541 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-model-cache\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.454244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.454219 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-dshm\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.454410 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.454389 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373ca1dd-aa36-4d69-af70-2970f27c1c97-tls-certs\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.511515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.511411 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r92m\" (UniqueName: \"kubernetes.io/projected/373ca1dd-aa36-4d69-af70-2970f27c1c97-kube-api-access-6r92m\") pod \"rate-test-model-506e6d-kserve-7c7cc6897c-skpbv\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.556544 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.556514 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:46.744643 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.744562 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" podUID="7b9401bf-3719-4419-a9f1-658e6ba190d1" containerName="payload-pre-processing" containerID="cri-o://65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586" gracePeriod=25 Nov 03 09:49:46.903870 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:49:46.903782 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:49:46.949115 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:46.949071 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv"] Nov 03 09:49:46.951884 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:49:46.951843 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod373ca1dd_aa36_4d69_af70_2970f27c1c97.slice/crio-72e78d994d2a12785061bf913af9fba624c218bb27326df17e2506668213e63d WatchSource:0}: Error finding container 72e78d994d2a12785061bf913af9fba624c218bb27326df17e2506668213e63d: Status 404 returned error can't find the container with id 72e78d994d2a12785061bf913af9fba624c218bb27326df17e2506668213e63d Nov 03 09:49:47.205183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.205104 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" event={"ID":"373ca1dd-aa36-4d69-af70-2970f27c1c97","Type":"ContainerStarted","Data":"de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb"} Nov 03 09:49:47.205183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.205146 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" event={"ID":"373ca1dd-aa36-4d69-af70-2970f27c1c97","Type":"ContainerStarted","Data":"72e78d994d2a12785061bf913af9fba624c218bb27326df17e2506668213e63d"} Nov 03 09:49:47.323561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.323506 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" podStartSLOduration=1.323490897 podStartE2EDuration="1.323490897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:49:47.320169918 +0000 UTC m=+4220.245112154" watchObservedRunningTime="2026-11-03 09:49:47.323490897 +0000 UTC m=+4220.248433108" Nov 03 09:49:47.557491 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.557442 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:47.611508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.611458 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f"] Nov 03 09:49:47.616368 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.616346 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.673963 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.673565 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmwjg\" (UniqueName: \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-kube-api-access-mmwjg\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.674287 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.674264 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-podinfo\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.674527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.674507 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istiod-ca-cert\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.674743 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.674724 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-token\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.674925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.674906 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-socket\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.675098 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.675079 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-envoy\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.675325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.675307 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-data\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.675533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.675516 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-credential-socket\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.675905 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.675797 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-certs\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.676207 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.676182 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f"] Nov 03 09:49:47.726204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.726177 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f8439f5b-openshift-default-dockercfg-728hr\"" Nov 03 09:49:47.778503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.777777 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-certs\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.778503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.777853 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmwjg\" (UniqueName: \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-kube-api-access-mmwjg\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.778503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.777898 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-podinfo\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.778503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.777942 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istiod-ca-cert\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.778503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.777992 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-token\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.778503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.778016 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-socket\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.778503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.778056 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-envoy\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.778503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.778137 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-data\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.778503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.778175 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-credential-socket\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.779319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.778643 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-credential-socket\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.779319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.778893 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-certs\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.779319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.779172 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-socket\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.784008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.780255 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-data\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.784008 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.780706 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istiod-ca-cert\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.784354 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.784307 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-podinfo\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.784528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.784465 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-envoy\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.838392 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.838302 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-token\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.838727 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.838704 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmwjg\" (UniqueName: \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-kube-api-access-mmwjg\") pod \"e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:47.931590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:47.931554 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:48.380984 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:48.380945 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f"] Nov 03 09:49:48.381669 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:49:48.381638 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186ae8ad_21e6_4fde_afaa_549164cc1b1d.slice/crio-9ee0e85db513251b8f17c24de6c677ed25e11f267e454cf809ac4770c3bf5afb WatchSource:0}: Error finding container 9ee0e85db513251b8f17c24de6c677ed25e11f267e454cf809ac4770c3bf5afb: Status 404 returned error can't find the container with id 9ee0e85db513251b8f17c24de6c677ed25e11f267e454cf809ac4770c3bf5afb Nov 03 09:49:48.385275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:48.385236 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:49:48.385366 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:48.385326 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:49:48.385407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:48.385367 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:49:48.593920 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:48.593885 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" podUID="7b9401bf-3719-4419-a9f1-658e6ba190d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:49:49.214967 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:49.214929 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" event={"ID":"186ae8ad-21e6-4fde-afaa-549164cc1b1d","Type":"ContainerStarted","Data":"35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf"} Nov 03 09:49:49.215137 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:49.214983 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" event={"ID":"186ae8ad-21e6-4fde-afaa-549164cc1b1d","Type":"ContainerStarted","Data":"9ee0e85db513251b8f17c24de6c677ed25e11f267e454cf809ac4770c3bf5afb"} Nov 03 09:49:49.334192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:49.334137 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" podStartSLOduration=2.334123372 podStartE2EDuration="2.334123372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:49:49.33355731 +0000 UTC m=+4222.258499529" watchObservedRunningTime="2026-11-03 09:49:49.334123372 +0000 UTC m=+4222.259065589" Nov 03 09:49:49.932151 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:49.932117 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:49.937912 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:49.937887 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:50.225370 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:50.225338 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:50.233534 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:50.233466 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:49:56.557384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:56.557333 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:56.570004 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:56.569969 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:57.288931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:57.288902 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:49:58.594188 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:49:58.594150 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" podUID="7b9401bf-3719-4419-a9f1-658e6ba190d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:50:07.025332 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.025267 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:50:07.072143 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.072109 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9401bf-3719-4419-a9f1-658e6ba190d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9401bf-3719-4419-a9f1-658e6ba190d1-plugins-config-volume\") pod \"7b9401bf-3719-4419-a9f1-658e6ba190d1\" (UID: \"7b9401bf-3719-4419-a9f1-658e6ba190d1\") " Nov 03 09:50:07.072340 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.072255 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9401bf-3719-4419-a9f1-658e6ba190d1-kube-api-access-7667l\" (UniqueName: \"kubernetes.io/projected/7b9401bf-3719-4419-a9f1-658e6ba190d1-kube-api-access-7667l\") pod \"7b9401bf-3719-4419-a9f1-658e6ba190d1\" (UID: \"7b9401bf-3719-4419-a9f1-658e6ba190d1\") " Nov 03 09:50:07.072663 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.072630 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9401bf-3719-4419-a9f1-658e6ba190d1-plugins-config-volume" pod "7b9401bf-3719-4419-a9f1-658e6ba190d1" (UID: "7b9401bf-3719-4419-a9f1-658e6ba190d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:50:07.073285 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.073171 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9401bf-3719-4419-a9f1-658e6ba190d1-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:07.075133 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.075017 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9401bf-3719-4419-a9f1-658e6ba190d1-kube-api-access-7667l" pod "7b9401bf-3719-4419-a9f1-658e6ba190d1" (UID: "7b9401bf-3719-4419-a9f1-658e6ba190d1"). InnerVolumeSpecName "kube-api-access-7667l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:50:07.174948 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.174905 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7667l\" (UniqueName: \"kubernetes.io/projected/7b9401bf-3719-4419-a9f1-658e6ba190d1-kube-api-access-7667l\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:07.357334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.357245 5015 generic.go:356] "Generic (PLEG): container finished" podID="7b9401bf-3719-4419-a9f1-658e6ba190d1" containerID="65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586" exitCode=0 Nov 03 09:50:07.357334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.357285 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" event={"ID":"7b9401bf-3719-4419-a9f1-658e6ba190d1","Type":"ContainerDied","Data":"65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586"} Nov 03 09:50:07.357334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.357322 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" event={"ID":"7b9401bf-3719-4419-a9f1-658e6ba190d1","Type":"ContainerDied","Data":"65165101b26e0c193ab50af614704412f5a7eadefb42f653c8dc96bfc4f53240"} Nov 03 09:50:07.357654 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.357343 5015 scope.go:122] "RemoveContainer" containerID="65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586" Nov 03 09:50:07.357654 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.357453 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs" Nov 03 09:50:07.370379 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.370309 5015 scope.go:122] "RemoveContainer" containerID="65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586" Nov 03 09:50:07.370790 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:50:07.370755 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586\": container with ID starting with 65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586 not found: ID does not exist" containerID="65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586" Nov 03 09:50:07.370895 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.370799 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586"} err="failed to get container status \"65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586\": rpc error: code = NotFound desc = could not find container \"65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586\": container with ID starting with 65a65194933bc5f164a90a47aaf01f22a07ab8f3f66ebe2b611db3f7b725b586 not found: ID does not exist" Nov 03 09:50:07.528090 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.528056 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs"] Nov 03 09:50:07.580861 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:07.580815 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3df1cc-74898558f5-pn9vs"] Nov 03 09:50:09.205784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:09.205753 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9401bf-3719-4419-a9f1-658e6ba190d1" path="/var/lib/kubelet/pods/7b9401bf-3719-4419-a9f1-658e6ba190d1/volumes" Nov 03 09:50:16.904684 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:50:16.904640 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:50:46.903356 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:50:46.903260 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:50:50.379568 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:50.379529 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw"] Nov 03 09:50:50.427716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:50.427668 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72"] Nov 03 09:50:50.431250 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:50:50.431222 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb"] Nov 03 09:51:04.933607 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:04.933575 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l"] Nov 03 09:51:05.045068 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.045035 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4"] Nov 03 09:51:05.403853 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.403813 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" containerID="cri-o://e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce" gracePeriod=45 Nov 03 09:51:05.453191 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.453126 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" containerID="cri-o://4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9" gracePeriod=45 Nov 03 09:51:05.454345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.454312 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" containerID="cri-o://daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136" gracePeriod=45 Nov 03 09:51:05.467194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.467165 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": dial tcp 10.132.0.93:8000: connect: connection refused" Nov 03 09:51:05.758342 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.757950 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 09:51:05.763267 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.763242 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 09:51:05.784409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.784348 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 09:51:05.833749 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.833713 5015 generic.go:356] "Generic (PLEG): container finished" podID="0b58da17-7e7e-4a49-8550-23803cb72517" containerID="daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136" exitCode=0 Nov 03 09:51:05.833947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.833778 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" Nov 03 09:51:05.833947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.833804 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" event={"ID":"0b58da17-7e7e-4a49-8550-23803cb72517","Type":"ContainerDied","Data":"daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136"} Nov 03 09:51:05.833947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.833862 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72" event={"ID":"0b58da17-7e7e-4a49-8550-23803cb72517","Type":"ContainerDied","Data":"7d8d36f0d3439b9ce1b70e3fe849dc591394369aa0c263eab7428ebc6c4ee45e"} Nov 03 09:51:05.833947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.833880 5015 scope.go:122] "RemoveContainer" containerID="daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136" Nov 03 09:51:05.836225 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.836198 5015 generic.go:356] "Generic (PLEG): container finished" podID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerID="4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9" exitCode=0 Nov 03 09:51:05.836408 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.836265 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" Nov 03 09:51:05.836408 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.836309 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" event={"ID":"fa5a15cb-ba0a-463c-81cb-e85ad047b627","Type":"ContainerDied","Data":"4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9"} Nov 03 09:51:05.836408 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.836383 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb" event={"ID":"fa5a15cb-ba0a-463c-81cb-e85ad047b627","Type":"ContainerDied","Data":"23939eda6367eb2a5140731aac2672487f1f39ea9156f7668a6f3160585ec9bf"} Nov 03 09:51:05.838132 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.838093 5015 generic.go:356] "Generic (PLEG): container finished" podID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerID="e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce" exitCode=0 Nov 03 09:51:05.838264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.838139 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" Nov 03 09:51:05.838264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.838182 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" event={"ID":"acffa72b-ec10-45dd-99fb-614a015f41ba","Type":"ContainerDied","Data":"e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce"} Nov 03 09:51:05.838264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.838208 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw" event={"ID":"acffa72b-ec10-45dd-99fb-614a015f41ba","Type":"ContainerDied","Data":"3b5cbfe3bf2e633c6f58b2f4d40853352cb79eddc625c03788efb4857d9023f8"} Nov 03 09:51:05.845727 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.845710 5015 scope.go:122] "RemoveContainer" containerID="daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136" Nov 03 09:51:05.846021 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:51:05.846001 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136\": container with ID starting with daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136 not found: ID does not exist" containerID="daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136" Nov 03 09:51:05.846089 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.846029 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136"} err="failed to get container status \"daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136\": rpc error: code = NotFound desc = could not find container \"daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136\": container with ID starting with daf0391be6fc0ed27a023ca8db7ad60c856aefd04a2ed9cbd68dc4fc5fd3d136 not found: ID does not exist" Nov 03 09:51:05.846089 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.846047 5015 scope.go:122] "RemoveContainer" containerID="4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9" Nov 03 09:51:05.846768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.846745 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5a15cb-ba0a-463c-81cb-e85ad047b627-kube-api-access-gk296\" (UniqueName: \"kubernetes.io/projected/fa5a15cb-ba0a-463c-81cb-e85ad047b627-kube-api-access-gk296\") pod \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " Nov 03 09:51:05.846855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.846812 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-dshm\") pod \"acffa72b-ec10-45dd-99fb-614a015f41ba\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " Nov 03 09:51:05.846855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.846834 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acffa72b-ec10-45dd-99fb-614a015f41ba-kube-api-access-p5kmw\" (UniqueName: \"kubernetes.io/projected/acffa72b-ec10-45dd-99fb-614a015f41ba-kube-api-access-p5kmw\") pod \"acffa72b-ec10-45dd-99fb-614a015f41ba\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " Nov 03 09:51:05.846947 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.846857 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acffa72b-ec10-45dd-99fb-614a015f41ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/acffa72b-ec10-45dd-99fb-614a015f41ba-tls-certs\") pod \"acffa72b-ec10-45dd-99fb-614a015f41ba\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " Nov 03 09:51:05.847007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.846982 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-model-cache\") pod \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " Nov 03 09:51:05.847297 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.847259 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-model-cache" pod "fa5a15cb-ba0a-463c-81cb-e85ad047b627" (UID: "fa5a15cb-ba0a-463c-81cb-e85ad047b627"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:05.847527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.847490 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-dshm\") pod \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " Nov 03 09:51:05.847838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.847811 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-tmp-dir\") pod \"acffa72b-ec10-45dd-99fb-614a015f41ba\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " Nov 03 09:51:05.847982 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.847901 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tmp-dir\") pod \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " Nov 03 09:51:05.848088 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848070 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tls-certs\") pod \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " Nov 03 09:51:05.848166 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848130 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-home\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-home\") pod \"acffa72b-ec10-45dd-99fb-614a015f41ba\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " Nov 03 09:51:05.848232 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848179 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-model-cache\") pod \"acffa72b-ec10-45dd-99fb-614a015f41ba\" (UID: \"acffa72b-ec10-45dd-99fb-614a015f41ba\") " Nov 03 09:51:05.848293 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848235 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-home\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-home\") pod \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\" (UID: \"fa5a15cb-ba0a-463c-81cb-e85ad047b627\") " Nov 03 09:51:05.848293 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848244 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tmp-dir" pod "fa5a15cb-ba0a-463c-81cb-e85ad047b627" (UID: "fa5a15cb-ba0a-463c-81cb-e85ad047b627"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:05.848736 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848654 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-model-cache" pod "acffa72b-ec10-45dd-99fb-614a015f41ba" (UID: "acffa72b-ec10-45dd-99fb-614a015f41ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:05.848736 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848706 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-tmp-dir" pod "acffa72b-ec10-45dd-99fb-614a015f41ba" (UID: "acffa72b-ec10-45dd-99fb-614a015f41ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:05.848898 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848830 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-home" pod "fa5a15cb-ba0a-463c-81cb-e85ad047b627" (UID: "fa5a15cb-ba0a-463c-81cb-e85ad047b627"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:05.848898 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848874 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-home" pod "acffa72b-ec10-45dd-99fb-614a015f41ba" (UID: "acffa72b-ec10-45dd-99fb-614a015f41ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:05.848898 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848893 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.849052 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848910 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.849052 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848937 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.849052 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.848949 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.850158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.850129 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-dshm" pod "fa5a15cb-ba0a-463c-81cb-e85ad047b627" (UID: "fa5a15cb-ba0a-463c-81cb-e85ad047b627"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:05.850439 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.850245 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acffa72b-ec10-45dd-99fb-614a015f41ba-tls-certs" pod "acffa72b-ec10-45dd-99fb-614a015f41ba" (UID: "acffa72b-ec10-45dd-99fb-614a015f41ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:51:05.850439 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.850311 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5a15cb-ba0a-463c-81cb-e85ad047b627-kube-api-access-gk296" pod "fa5a15cb-ba0a-463c-81cb-e85ad047b627" (UID: "fa5a15cb-ba0a-463c-81cb-e85ad047b627"). InnerVolumeSpecName "kube-api-access-gk296". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:51:05.850439 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.850363 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acffa72b-ec10-45dd-99fb-614a015f41ba-kube-api-access-p5kmw" pod "acffa72b-ec10-45dd-99fb-614a015f41ba" (UID: "acffa72b-ec10-45dd-99fb-614a015f41ba"). InnerVolumeSpecName "kube-api-access-p5kmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:51:05.851023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.850997 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tls-certs" pod "fa5a15cb-ba0a-463c-81cb-e85ad047b627" (UID: "fa5a15cb-ba0a-463c-81cb-e85ad047b627"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:51:05.851792 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.851772 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-dshm" pod "acffa72b-ec10-45dd-99fb-614a015f41ba" (UID: "acffa72b-ec10-45dd-99fb-614a015f41ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:05.856717 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.856694 5015 scope.go:122] "RemoveContainer" containerID="4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9" Nov 03 09:51:05.857000 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:51:05.856982 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9\": container with ID starting with 4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9 not found: ID does not exist" containerID="4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9" Nov 03 09:51:05.857075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.857006 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9"} err="failed to get container status \"4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9\": rpc error: code = NotFound desc = could not find container \"4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9\": container with ID starting with 4f535c935e2eb620b5d860a8e3754208f7c6d65fa8234210ea481a530f0ea6c9 not found: ID does not exist" Nov 03 09:51:05.857075 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.857025 5015 scope.go:122] "RemoveContainer" containerID="e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce" Nov 03 09:51:05.866346 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.866319 5015 scope.go:122] "RemoveContainer" containerID="e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce" Nov 03 09:51:05.866787 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:51:05.866765 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce\": container with ID starting with e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce not found: ID does not exist" containerID="e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce" Nov 03 09:51:05.866857 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.866792 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce"} err="failed to get container status \"e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce\": rpc error: code = NotFound desc = could not find container \"e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce\": container with ID starting with e9229cecd1ea01b2e67b278fd7b3cd7f1cc9bc52259569960bae7d881d1963ce not found: ID does not exist" Nov 03 09:51:05.949671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.949591 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b58da17-7e7e-4a49-8550-23803cb72517-kube-api-access-phbfj\" (UniqueName: \"kubernetes.io/projected/0b58da17-7e7e-4a49-8550-23803cb72517-kube-api-access-phbfj\") pod \"0b58da17-7e7e-4a49-8550-23803cb72517\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " Nov 03 09:51:05.949671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.949644 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-home\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-home\") pod \"0b58da17-7e7e-4a49-8550-23803cb72517\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " Nov 03 09:51:05.950174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.949807 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-tmp-dir\") pod \"0b58da17-7e7e-4a49-8550-23803cb72517\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " Nov 03 09:51:05.950174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.949848 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-model-cache\") pod \"0b58da17-7e7e-4a49-8550-23803cb72517\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " Nov 03 09:51:05.950174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.949941 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b58da17-7e7e-4a49-8550-23803cb72517-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b58da17-7e7e-4a49-8550-23803cb72517-tls-certs\") pod \"0b58da17-7e7e-4a49-8550-23803cb72517\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " Nov 03 09:51:05.950174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.949984 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-home" pod "0b58da17-7e7e-4a49-8550-23803cb72517" (UID: "0b58da17-7e7e-4a49-8550-23803cb72517"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:05.950174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950030 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-dshm\") pod \"0b58da17-7e7e-4a49-8550-23803cb72517\" (UID: \"0b58da17-7e7e-4a49-8550-23803cb72517\") " Nov 03 09:51:05.950174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950048 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-tmp-dir" pod "0b58da17-7e7e-4a49-8550-23803cb72517" (UID: "0b58da17-7e7e-4a49-8550-23803cb72517"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:05.950174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950139 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-model-cache" pod "0b58da17-7e7e-4a49-8550-23803cb72517" (UID: "0b58da17-7e7e-4a49-8550-23803cb72517"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:05.950645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950585 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5a15cb-ba0a-463c-81cb-e85ad047b627-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.950645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950608 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.950645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950622 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.950812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950681 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.950877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950810 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk296\" (UniqueName: \"kubernetes.io/projected/fa5a15cb-ba0a-463c-81cb-e85ad047b627-kube-api-access-gk296\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.950877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950824 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.950877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950838 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acffa72b-ec10-45dd-99fb-614a015f41ba-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.950877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950850 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5kmw\" (UniqueName: \"kubernetes.io/projected/acffa72b-ec10-45dd-99fb-614a015f41ba-kube-api-access-p5kmw\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.950877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950867 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acffa72b-ec10-45dd-99fb-614a015f41ba-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950880 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5a15cb-ba0a-463c-81cb-e85ad047b627-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.951327 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.950892 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:05.952039 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.952015 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b58da17-7e7e-4a49-8550-23803cb72517-kube-api-access-phbfj" pod "0b58da17-7e7e-4a49-8550-23803cb72517" (UID: "0b58da17-7e7e-4a49-8550-23803cb72517"). InnerVolumeSpecName "kube-api-access-phbfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:51:05.952251 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.952234 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b58da17-7e7e-4a49-8550-23803cb72517-tls-certs" pod "0b58da17-7e7e-4a49-8550-23803cb72517" (UID: "0b58da17-7e7e-4a49-8550-23803cb72517"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:51:05.952778 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:05.952755 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-dshm" pod "0b58da17-7e7e-4a49-8550-23803cb72517" (UID: "0b58da17-7e7e-4a49-8550-23803cb72517"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:51:06.052243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:06.052205 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b58da17-7e7e-4a49-8550-23803cb72517-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:06.052243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:06.052240 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phbfj\" (UniqueName: \"kubernetes.io/projected/0b58da17-7e7e-4a49-8550-23803cb72517-kube-api-access-phbfj\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:06.052464 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:06.052252 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b58da17-7e7e-4a49-8550-23803cb72517-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:06.289358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:06.289323 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw"] Nov 03 09:51:06.293158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:06.293128 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-w1-26e861-kserve-69899c5848-4phjw"] Nov 03 09:51:06.411730 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:06.411694 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb"] Nov 03 09:51:06.415269 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:06.415242 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/simulator-w1-26e861-kserve-5b95ccf86d-qvstb"] Nov 03 09:51:06.523620 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:06.523587 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72"] Nov 03 09:51:06.529180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:06.529150 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/premium-w1-26e861-kserve-5bbdc49f65-tpd72"] Nov 03 09:51:07.204629 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:07.204596 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" path="/var/lib/kubelet/pods/0b58da17-7e7e-4a49-8550-23803cb72517/volumes" Nov 03 09:51:07.205094 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:07.205078 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" path="/var/lib/kubelet/pods/acffa72b-ec10-45dd-99fb-614a015f41ba/volumes" Nov 03 09:51:07.205505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:07.205468 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" path="/var/lib/kubelet/pods/fa5a15cb-ba0a-463c-81cb-e85ad047b627/volumes" Nov 03 09:51:09.934254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:09.934213 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" podUID="eff24799-01ca-45f0-8370-b5158f0c258a" containerName="payload-processing" containerID="cri-o://bed54b8ac55c1d7ba18d1e1685a2882c8e4901e9d242215243ba911b54b5e8a9" gracePeriod=25 Nov 03 09:51:10.046053 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:10.045993 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" podUID="efe094fc-cbb3-4e65-9f1f-2fc328584884" containerName="payload-pre-processing" containerID="cri-o://791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233" gracePeriod=25 Nov 03 09:51:14.485975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:14.485930 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" podUID="eff24799-01ca-45f0-8370-b5158f0c258a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:51:14.486430 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:14.486091 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" podUID="efe094fc-cbb3-4e65-9f1f-2fc328584884" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:51:16.902635 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:51:16.902595 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:51:24.485592 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:24.485545 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" podUID="eff24799-01ca-45f0-8370-b5158f0c258a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:51:24.485592 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:24.485586 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" podUID="efe094fc-cbb3-4e65-9f1f-2fc328584884" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:51:30.065203 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.065169 5015 generic.go:356] "Generic (PLEG): container finished" podID="eff24799-01ca-45f0-8370-b5158f0c258a" containerID="bed54b8ac55c1d7ba18d1e1685a2882c8e4901e9d242215243ba911b54b5e8a9" exitCode=0 Nov 03 09:51:30.065610 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.065246 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" event={"ID":"eff24799-01ca-45f0-8370-b5158f0c258a","Type":"ContainerDied","Data":"bed54b8ac55c1d7ba18d1e1685a2882c8e4901e9d242215243ba911b54b5e8a9"} Nov 03 09:51:30.067375 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.067345 5015 generic.go:356] "Generic (PLEG): container finished" podID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerID="35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf" exitCode=0 Nov 03 09:51:30.067589 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.067504 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" event={"ID":"186ae8ad-21e6-4fde-afaa-549164cc1b1d","Type":"ContainerDied","Data":"35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf"} Nov 03 09:51:30.068042 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.068010 5015 scope.go:122] "RemoveContainer" containerID="35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf" Nov 03 09:51:30.070037 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.070003 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:51:30.070129 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.070092 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:51:30.070189 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.070165 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:51:30.268604 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.268513 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 09:51:30.396018 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.394951 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 09:51:30.396018 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.395571 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff24799-01ca-45f0-8370-b5158f0c258a-kube-api-access-b6t47\" (UniqueName: \"kubernetes.io/projected/eff24799-01ca-45f0-8370-b5158f0c258a-kube-api-access-b6t47\") pod \"eff24799-01ca-45f0-8370-b5158f0c258a\" (UID: \"eff24799-01ca-45f0-8370-b5158f0c258a\") " Nov 03 09:51:30.396018 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.395627 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff24799-01ca-45f0-8370-b5158f0c258a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff24799-01ca-45f0-8370-b5158f0c258a-plugins-config-volume\") pod \"eff24799-01ca-45f0-8370-b5158f0c258a\" (UID: \"eff24799-01ca-45f0-8370-b5158f0c258a\") " Nov 03 09:51:30.396436 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.396402 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff24799-01ca-45f0-8370-b5158f0c258a-plugins-config-volume" pod "eff24799-01ca-45f0-8370-b5158f0c258a" (UID: "eff24799-01ca-45f0-8370-b5158f0c258a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:51:30.398371 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.398346 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff24799-01ca-45f0-8370-b5158f0c258a-kube-api-access-b6t47" pod "eff24799-01ca-45f0-8370-b5158f0c258a" (UID: "eff24799-01ca-45f0-8370-b5158f0c258a"). InnerVolumeSpecName "kube-api-access-b6t47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:51:30.496682 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.496646 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efe094fc-cbb3-4e65-9f1f-2fc328584884-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe094fc-cbb3-4e65-9f1f-2fc328584884-plugins-config-volume\") pod \"efe094fc-cbb3-4e65-9f1f-2fc328584884\" (UID: \"efe094fc-cbb3-4e65-9f1f-2fc328584884\") " Nov 03 09:51:30.496904 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.496775 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe094fc-cbb3-4e65-9f1f-2fc328584884-kube-api-access-5sb5w\" (UniqueName: \"kubernetes.io/projected/efe094fc-cbb3-4e65-9f1f-2fc328584884-kube-api-access-5sb5w\") pod \"efe094fc-cbb3-4e65-9f1f-2fc328584884\" (UID: \"efe094fc-cbb3-4e65-9f1f-2fc328584884\") " Nov 03 09:51:30.497127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.497093 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe094fc-cbb3-4e65-9f1f-2fc328584884-plugins-config-volume" pod "efe094fc-cbb3-4e65-9f1f-2fc328584884" (UID: "efe094fc-cbb3-4e65-9f1f-2fc328584884"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:51:30.497298 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.497267 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe094fc-cbb3-4e65-9f1f-2fc328584884-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:30.497298 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.497294 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6t47\" (UniqueName: \"kubernetes.io/projected/eff24799-01ca-45f0-8370-b5158f0c258a-kube-api-access-b6t47\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:30.497513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.497309 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff24799-01ca-45f0-8370-b5158f0c258a-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:30.499280 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.499253 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe094fc-cbb3-4e65-9f1f-2fc328584884-kube-api-access-5sb5w" pod "efe094fc-cbb3-4e65-9f1f-2fc328584884" (UID: "efe094fc-cbb3-4e65-9f1f-2fc328584884"). InnerVolumeSpecName "kube-api-access-5sb5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:51:30.598494 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:30.598450 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sb5w\" (UniqueName: \"kubernetes.io/projected/efe094fc-cbb3-4e65-9f1f-2fc328584884-kube-api-access-5sb5w\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:51:31.075578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.075454 5015 generic.go:356] "Generic (PLEG): container finished" podID="efe094fc-cbb3-4e65-9f1f-2fc328584884" containerID="791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233" exitCode=0 Nov 03 09:51:31.075578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.075531 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" event={"ID":"efe094fc-cbb3-4e65-9f1f-2fc328584884","Type":"ContainerDied","Data":"791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233"} Nov 03 09:51:31.076076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.075541 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" Nov 03 09:51:31.076076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.075639 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4" event={"ID":"efe094fc-cbb3-4e65-9f1f-2fc328584884","Type":"ContainerDied","Data":"49d340e1dc5678a3eba9c0b0775268d93d52eaaa8218280f7e03aa4131a41faa"} Nov 03 09:51:31.076076 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.075673 5015 scope.go:122] "RemoveContainer" containerID="791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233" Nov 03 09:51:31.077588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.077560 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" event={"ID":"eff24799-01ca-45f0-8370-b5158f0c258a","Type":"ContainerDied","Data":"c3c1e277de91b10a9db49f1b3e7d0c300aeefbe75b155437d64a7fba500d2a67"} Nov 03 09:51:31.077713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.077685 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l" Nov 03 09:51:31.079832 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.079802 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" event={"ID":"186ae8ad-21e6-4fde-afaa-549164cc1b1d","Type":"ContainerStarted","Data":"033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb"} Nov 03 09:51:31.088511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.088495 5015 scope.go:122] "RemoveContainer" containerID="791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233" Nov 03 09:51:31.088823 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:51:31.088804 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233\": container with ID starting with 791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233 not found: ID does not exist" containerID="791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233" Nov 03 09:51:31.088881 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.088832 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233"} err="failed to get container status \"791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233\": rpc error: code = NotFound desc = could not find container \"791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233\": container with ID starting with 791d35e6212ec57c35ee564ddd5d80701d96d12caa254738e527523900d32233 not found: ID does not exist" Nov 03 09:51:31.088881 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.088851 5015 scope.go:122] "RemoveContainer" containerID="bed54b8ac55c1d7ba18d1e1685a2882c8e4901e9d242215243ba911b54b5e8a9" Nov 03 09:51:31.263585 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.263535 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4"] Nov 03 09:51:31.305911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.305878 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-26e861-6c6959d97f-phqf4"] Nov 03 09:51:31.424363 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.424271 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l"] Nov 03 09:51:31.432810 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.432780 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26e861-7555fb8649-9h44l"] Nov 03 09:51:31.932018 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.931982 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:51:31.933268 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.933237 5015 patch_prober.go:29] interesting pod/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 09:51:31.933403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:31.933293 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 09:51:32.932725 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:32.932684 5015 patch_prober.go:29] interesting pod/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 09:51:32.933296 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:32.932766 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 09:51:33.204933 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:33.204857 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe094fc-cbb3-4e65-9f1f-2fc328584884" path="/var/lib/kubelet/pods/efe094fc-cbb3-4e65-9f1f-2fc328584884/volumes" Nov 03 09:51:33.205303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:33.205290 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff24799-01ca-45f0-8370-b5158f0c258a" path="/var/lib/kubelet/pods/eff24799-01ca-45f0-8370-b5158f0c258a/volumes" Nov 03 09:51:33.932201 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:33.932162 5015 patch_prober.go:29] interesting pod/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 09:51:33.932400 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:33.932230 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 09:51:34.932687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:34.932649 5015 patch_prober.go:29] interesting pod/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 09:51:34.933115 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:34.932705 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 09:51:35.225955 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:35.225931 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:51:35.932104 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:35.932067 5015 patch_prober.go:29] interesting pod/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 09:51:35.932288 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:35.932121 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 09:51:36.937636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:36.937604 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:51:36.938943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:36.938916 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:51:42.893461 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.893424 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs"] Nov 03 09:51:42.894062 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894048 5015 cpu_manager.go:414] "containerMap: removing container" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" Nov 03 09:51:42.894112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894066 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" Nov 03 09:51:42.894112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894080 5015 cpu_manager.go:414] "containerMap: removing container" podUID="eff24799-01ca-45f0-8370-b5158f0c258a" containerName="payload-processing" Nov 03 09:51:42.894112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894085 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff24799-01ca-45f0-8370-b5158f0c258a" containerName="payload-processing" Nov 03 09:51:42.894112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894092 5015 cpu_manager.go:414] "containerMap: removing container" podUID="efe094fc-cbb3-4e65-9f1f-2fc328584884" containerName="payload-pre-processing" Nov 03 09:51:42.894112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894097 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe094fc-cbb3-4e65-9f1f-2fc328584884" containerName="payload-pre-processing" Nov 03 09:51:42.894112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894114 5015 cpu_manager.go:414] "containerMap: removing container" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" Nov 03 09:51:42.894355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894118 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" Nov 03 09:51:42.894355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894136 5015 cpu_manager.go:414] "containerMap: removing container" podUID="7b9401bf-3719-4419-a9f1-658e6ba190d1" containerName="payload-pre-processing" Nov 03 09:51:42.894355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894141 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9401bf-3719-4419-a9f1-658e6ba190d1" containerName="payload-pre-processing" Nov 03 09:51:42.894355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894156 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" Nov 03 09:51:42.894355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894160 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" Nov 03 09:51:42.894355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894241 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5a15cb-ba0a-463c-81cb-e85ad047b627" containerName="main" Nov 03 09:51:42.894355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894249 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff24799-01ca-45f0-8370-b5158f0c258a" containerName="payload-processing" Nov 03 09:51:42.894355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894256 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe094fc-cbb3-4e65-9f1f-2fc328584884" containerName="payload-pre-processing" Nov 03 09:51:42.894355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894265 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="acffa72b-ec10-45dd-99fb-614a015f41ba" containerName="main" Nov 03 09:51:42.894355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894274 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9401bf-3719-4419-a9f1-658e6ba190d1" containerName="payload-pre-processing" Nov 03 09:51:42.894355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.894282 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b58da17-7e7e-4a49-8550-23803cb72517" containerName="main" Nov 03 09:51:42.896627 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.896611 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:42.951426 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.951383 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-26e861-openshift-default-dockercfg-4wl7s\"" Nov 03 09:51:42.981767 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.981726 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:42.981767 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.981770 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:42.982043 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.981929 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:42.982043 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.981981 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:42.982153 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.982068 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hwz\" (UniqueName: \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-kube-api-access-g4hwz\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:42.982219 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.982162 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:42.982219 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.982203 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:42.982320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.982234 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:42.982358 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:42.982312 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.006541 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.006503 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs"] Nov 03 09:51:43.083968 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.083900 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.084742 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.084710 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.085407 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.085387 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.086022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.084627 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.086503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.086414 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.086742 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.085946 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.086921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.086901 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hwz\" (UniqueName: \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-kube-api-access-g4hwz\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.087378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.087357 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.087548 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.087531 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.087693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.087675 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.087824 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.087799 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.087971 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.087955 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.088159 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.088135 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.088843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.088817 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.125070 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.123988 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.125070 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.124613 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.147589 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.147467 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.147734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.147607 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hwz\" (UniqueName: \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-kube-api-access-g4hwz\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.214271 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.213316 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:43.666048 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.666009 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs"] Nov 03 09:51:43.667574 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:51:43.667543 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228c6eb5_fcf2_422e_a4a6_19a6043bbe21.slice/crio-a0e2279e1b415c55520e943f94ca76b37f27c667d3b7715511209766b2cdd358 WatchSource:0}: Error finding container a0e2279e1b415c55520e943f94ca76b37f27c667d3b7715511209766b2cdd358: Status 404 returned error can't find the container with id a0e2279e1b415c55520e943f94ca76b37f27c667d3b7715511209766b2cdd358 Nov 03 09:51:43.670464 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.670425 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:51:43.670553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.670536 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:51:43.670594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:43.670582 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:51:44.189736 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:44.189698 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" event={"ID":"228c6eb5-fcf2-422e-a4a6-19a6043bbe21","Type":"ContainerStarted","Data":"c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d"} Nov 03 09:51:44.190172 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:44.189758 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" event={"ID":"228c6eb5-fcf2-422e-a4a6-19a6043bbe21","Type":"ContainerStarted","Data":"a0e2279e1b415c55520e943f94ca76b37f27c667d3b7715511209766b2cdd358"} Nov 03 09:51:44.218286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:44.214841 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:44.220645 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:44.220609 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 09:51:44.220838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:44.220663 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" podUID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 09:51:44.319247 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:44.319178 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" podStartSLOduration=2.319157893 podStartE2EDuration="2.319157893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:51:44.312021625 +0000 UTC m=+4337.236963841" watchObservedRunningTime="2026-11-03 09:51:44.319157893 +0000 UTC m=+4337.244100110" Nov 03 09:51:45.214465 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:45.214432 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 09:51:45.214975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:45.214507 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" podUID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 09:51:46.214226 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:46.214187 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 09:51:46.214416 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:46.214238 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" podUID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 09:51:46.904522 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:51:46.904463 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:51:47.214124 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:47.214046 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 09:51:47.214124 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:47.214107 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" podUID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 09:51:48.213812 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:48.213779 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 09:51:48.214237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:48.213833 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" podUID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 09:51:49.214405 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:49.214369 5015 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 09:51:49.214825 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:49.214430 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" podUID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 09:51:50.218991 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:50.218958 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:50.219409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:50.219253 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:51:50.219876 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:51:50.219855 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:52:16.904203 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:52:16.904117 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:52:26.730619 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:26.730587 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs"] Nov 03 09:52:26.731053 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:26.730805 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" podUID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" containerName="istio-proxy" containerID="cri-o://c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d" gracePeriod=30 Nov 03 09:52:32.001979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.001921 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:52:32.088334 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.088304 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-envoy\") pod \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " Nov 03 09:52:32.088564 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.088346 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-kube-api-access-g4hwz\" (UniqueName: \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-kube-api-access-g4hwz\") pod \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " Nov 03 09:52:32.088564 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.088371 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-podinfo\") pod \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " Nov 03 09:52:32.088564 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.088436 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-data\") pod \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " Nov 03 09:52:32.088564 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.088522 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-token\" (UniqueName: \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-token\") pod \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " Nov 03 09:52:32.088564 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.088560 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-credential-socket\") pod \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " Nov 03 09:52:32.088870 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.088606 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-certs\") pod \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " Nov 03 09:52:32.088870 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.088686 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istiod-ca-cert\") pod \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " Nov 03 09:52:32.088870 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.088736 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-socket\") pod \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\" (UID: \"228c6eb5-fcf2-422e-a4a6-19a6043bbe21\") " Nov 03 09:52:32.089028 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.088972 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-certs" pod "228c6eb5-fcf2-422e-a4a6-19a6043bbe21" (UID: "228c6eb5-fcf2-422e-a4a6-19a6043bbe21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:52:32.089241 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.089179 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-credential-socket" pod "228c6eb5-fcf2-422e-a4a6-19a6043bbe21" (UID: "228c6eb5-fcf2-422e-a4a6-19a6043bbe21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:52:32.089241 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.089214 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istiod-ca-cert" pod "228c6eb5-fcf2-422e-a4a6-19a6043bbe21" (UID: "228c6eb5-fcf2-422e-a4a6-19a6043bbe21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:52:32.089403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.089361 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-socket" pod "228c6eb5-fcf2-422e-a4a6-19a6043bbe21" (UID: "228c6eb5-fcf2-422e-a4a6-19a6043bbe21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:52:32.089565 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.089543 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.089636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.089569 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.089636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.089583 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.089636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.089596 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.090658 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.090630 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-data" pod "228c6eb5-fcf2-422e-a4a6-19a6043bbe21" (UID: "228c6eb5-fcf2-422e-a4a6-19a6043bbe21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:52:32.091101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.091073 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-kube-api-access-g4hwz" pod "228c6eb5-fcf2-422e-a4a6-19a6043bbe21" (UID: "228c6eb5-fcf2-422e-a4a6-19a6043bbe21"). InnerVolumeSpecName "kube-api-access-g4hwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:52:32.091256 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.091229 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-podinfo" pod "228c6eb5-fcf2-422e-a4a6-19a6043bbe21" (UID: "228c6eb5-fcf2-422e-a4a6-19a6043bbe21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:52:32.091707 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.091684 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-envoy" pod "228c6eb5-fcf2-422e-a4a6-19a6043bbe21" (UID: "228c6eb5-fcf2-422e-a4a6-19a6043bbe21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:52:32.092186 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.092166 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-token" pod "228c6eb5-fcf2-422e-a4a6-19a6043bbe21" (UID: "228c6eb5-fcf2-422e-a4a6-19a6043bbe21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:52:32.190913 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.190879 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.190913 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.190907 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4hwz\" (UniqueName: \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-kube-api-access-g4hwz\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.190913 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.190917 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.191168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.190925 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.191168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.190937 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228c6eb5-fcf2-422e-a4a6-19a6043bbe21-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.620594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.620557 5015 generic.go:356] "Generic (PLEG): container finished" podID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" containerID="c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d" exitCode=0 Nov 03 09:52:32.620894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.620610 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" Nov 03 09:52:32.620894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.620629 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" event={"ID":"228c6eb5-fcf2-422e-a4a6-19a6043bbe21","Type":"ContainerDied","Data":"c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d"} Nov 03 09:52:32.620894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.620667 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs" event={"ID":"228c6eb5-fcf2-422e-a4a6-19a6043bbe21","Type":"ContainerDied","Data":"a0e2279e1b415c55520e943f94ca76b37f27c667d3b7715511209766b2cdd358"} Nov 03 09:52:32.620894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.620689 5015 scope.go:122] "RemoveContainer" containerID="c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d" Nov 03 09:52:32.634822 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.634802 5015 scope.go:122] "RemoveContainer" containerID="c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d" Nov 03 09:52:32.635109 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:52:32.635092 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d\": container with ID starting with c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d not found: ID does not exist" containerID="c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d" Nov 03 09:52:32.635194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.635117 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d"} err="failed to get container status \"c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d\": rpc error: code = NotFound desc = could not find container \"c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d\": container with ID starting with c0bd89e49b8b4e9a15a7f35c3e0e2d6e3ed541d2684d70bb7b036245a144075d not found: ID does not exist" Nov 03 09:52:32.669882 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:52:32.669855 5015 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228c6eb5_fcf2_422e_a4a6_19a6043bbe21.slice/crio-a0e2279e1b415c55520e943f94ca76b37f27c667d3b7715511209766b2cdd358\": RecentStats: unable to find data in memory cache]" Nov 03 09:52:32.792579 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.792539 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs"] Nov 03 09:52:32.797082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:32.797058 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-k6ljs"] Nov 03 09:52:33.204786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:52:33.204752 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" path="/var/lib/kubelet/pods/228c6eb5-fcf2-422e-a4a6-19a6043bbe21/volumes" Nov 03 09:52:46.902945 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:52:46.902902 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:53:16.903402 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:53:16.903360 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:53:46.909365 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:53:46.909265 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:53:56.098665 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:53:56.098624 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn"] Nov 03 09:54:10.082199 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:10.082162 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f"] Nov 03 09:54:10.082840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:10.082442 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" containerID="cri-o://033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb" gracePeriod=30 Nov 03 09:54:11.120150 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.120101 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" podUID="384591ef-c634-4c99-8e9b-4f85332e5802" containerName="main" containerID="cri-o://ccfcf9681929097698e858fa30edc139bd4915e85b46e26a5c56e29b2d02e6f0" gracePeriod=45 Nov 03 09:54:11.396937 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.396908 5015 generic.go:356] "Generic (PLEG): container finished" podID="384591ef-c634-4c99-8e9b-4f85332e5802" containerID="ccfcf9681929097698e858fa30edc139bd4915e85b46e26a5c56e29b2d02e6f0" exitCode=0 Nov 03 09:54:11.397080 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.396976 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" event={"ID":"384591ef-c634-4c99-8e9b-4f85332e5802","Type":"ContainerDied","Data":"ccfcf9681929097698e858fa30edc139bd4915e85b46e26a5c56e29b2d02e6f0"} Nov 03 09:54:11.397080 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.397010 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" event={"ID":"384591ef-c634-4c99-8e9b-4f85332e5802","Type":"ContainerDied","Data":"e944680373b4aeb6cfb0fdc23edb6e0f198f352547c51aa77e0536fb2b5e3ee2"} Nov 03 09:54:11.397080 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.397022 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e944680373b4aeb6cfb0fdc23edb6e0f198f352547c51aa77e0536fb2b5e3ee2" Nov 03 09:54:11.408751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.408727 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:54:11.591192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.591153 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-model-cache\") pod \"384591ef-c634-4c99-8e9b-4f85332e5802\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " Nov 03 09:54:11.591505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.591460 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-tmp-dir\") pod \"384591ef-c634-4c99-8e9b-4f85332e5802\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " Nov 03 09:54:11.591752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.591731 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384591ef-c634-4c99-8e9b-4f85332e5802-kube-api-access-hbp95\" (UniqueName: \"kubernetes.io/projected/384591ef-c634-4c99-8e9b-4f85332e5802-kube-api-access-hbp95\") pod \"384591ef-c634-4c99-8e9b-4f85332e5802\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " Nov 03 09:54:11.591959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.591942 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/384591ef-c634-4c99-8e9b-4f85332e5802-tls-certs\" (UniqueName: \"kubernetes.io/secret/384591ef-c634-4c99-8e9b-4f85332e5802-tls-certs\") pod \"384591ef-c634-4c99-8e9b-4f85332e5802\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " Nov 03 09:54:11.592220 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.591749 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-model-cache" pod "384591ef-c634-4c99-8e9b-4f85332e5802" (UID: "384591ef-c634-4c99-8e9b-4f85332e5802"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:54:11.592220 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.591963 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-tmp-dir" pod "384591ef-c634-4c99-8e9b-4f85332e5802" (UID: "384591ef-c634-4c99-8e9b-4f85332e5802"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:54:11.592220 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.592160 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-home\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-home\") pod \"384591ef-c634-4c99-8e9b-4f85332e5802\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " Nov 03 09:54:11.592564 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.592384 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-home" pod "384591ef-c634-4c99-8e9b-4f85332e5802" (UID: "384591ef-c634-4c99-8e9b-4f85332e5802"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:54:11.592805 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.592780 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-dshm\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-dshm\") pod \"384591ef-c634-4c99-8e9b-4f85332e5802\" (UID: \"384591ef-c634-4c99-8e9b-4f85332e5802\") " Nov 03 09:54:11.594521 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.594491 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384591ef-c634-4c99-8e9b-4f85332e5802-kube-api-access-hbp95" pod "384591ef-c634-4c99-8e9b-4f85332e5802" (UID: "384591ef-c634-4c99-8e9b-4f85332e5802"). InnerVolumeSpecName "kube-api-access-hbp95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:54:11.598527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.594995 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:11.598527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.595023 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:11.598527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.595037 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:11.598527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.595052 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbp95\" (UniqueName: \"kubernetes.io/projected/384591ef-c634-4c99-8e9b-4f85332e5802-kube-api-access-hbp95\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:11.598527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.595211 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-dshm" pod "384591ef-c634-4c99-8e9b-4f85332e5802" (UID: "384591ef-c634-4c99-8e9b-4f85332e5802"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:54:11.598527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.595248 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/384591ef-c634-4c99-8e9b-4f85332e5802-tls-certs" pod "384591ef-c634-4c99-8e9b-4f85332e5802" (UID: "384591ef-c634-4c99-8e9b-4f85332e5802"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:54:11.696421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.696334 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/384591ef-c634-4c99-8e9b-4f85332e5802-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:11.696421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:11.696362 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/384591ef-c634-4c99-8e9b-4f85332e5802-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:12.403894 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:12.403865 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn" Nov 03 09:54:12.577093 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:12.577061 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn"] Nov 03 09:54:12.627310 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:12.627273 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/rate-test-model-433821-kserve-5798c67fc7-bfmqn"] Nov 03 09:54:13.204721 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:13.204689 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384591ef-c634-4c99-8e9b-4f85332e5802" path="/var/lib/kubelet/pods/384591ef-c634-4c99-8e9b-4f85332e5802/volumes" Nov 03 09:54:15.386314 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.386289 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:54:15.426057 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.426022 5015 generic.go:356] "Generic (PLEG): container finished" podID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerID="033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb" exitCode=0 Nov 03 09:54:15.426229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.426077 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" Nov 03 09:54:15.426229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.426112 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" event={"ID":"186ae8ad-21e6-4fde-afaa-549164cc1b1d","Type":"ContainerDied","Data":"033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb"} Nov 03 09:54:15.426229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.426174 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f" event={"ID":"186ae8ad-21e6-4fde-afaa-549164cc1b1d","Type":"ContainerDied","Data":"9ee0e85db513251b8f17c24de6c677ed25e11f267e454cf809ac4770c3bf5afb"} Nov 03 09:54:15.426229 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.426197 5015 scope.go:122] "RemoveContainer" containerID="033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb" Nov 03 09:54:15.437182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.437156 5015 scope.go:122] "RemoveContainer" containerID="35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf" Nov 03 09:54:15.444384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.443191 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-token\" (UniqueName: \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-token\") pod \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " Nov 03 09:54:15.444384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.443303 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-socket\") pod \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " Nov 03 09:54:15.444384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.443356 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-envoy\") pod \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " Nov 03 09:54:15.444384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.443401 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-certs\") pod \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " Nov 03 09:54:15.444384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.443498 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istiod-ca-cert\") pod \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " Nov 03 09:54:15.444384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.443537 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-credential-socket\") pod \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " Nov 03 09:54:15.444384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.443585 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-podinfo\") pod \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " Nov 03 09:54:15.444384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.443659 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-kube-api-access-mmwjg\" (UniqueName: \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-kube-api-access-mmwjg\") pod \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " Nov 03 09:54:15.444384 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.443690 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-data\") pod \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\" (UID: \"186ae8ad-21e6-4fde-afaa-549164cc1b1d\") " Nov 03 09:54:15.445079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.445049 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-data" pod "186ae8ad-21e6-4fde-afaa-549164cc1b1d" (UID: "186ae8ad-21e6-4fde-afaa-549164cc1b1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:54:15.445739 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.445700 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istiod-ca-cert" pod "186ae8ad-21e6-4fde-afaa-549164cc1b1d" (UID: "186ae8ad-21e6-4fde-afaa-549164cc1b1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:54:15.446002 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.445891 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-socket" pod "186ae8ad-21e6-4fde-afaa-549164cc1b1d" (UID: "186ae8ad-21e6-4fde-afaa-549164cc1b1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:54:15.446120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.446050 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-certs" pod "186ae8ad-21e6-4fde-afaa-549164cc1b1d" (UID: "186ae8ad-21e6-4fde-afaa-549164cc1b1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:54:15.446246 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.446208 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-credential-socket" pod "186ae8ad-21e6-4fde-afaa-549164cc1b1d" (UID: "186ae8ad-21e6-4fde-afaa-549164cc1b1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:54:15.449306 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.449260 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-kube-api-access-mmwjg" pod "186ae8ad-21e6-4fde-afaa-549164cc1b1d" (UID: "186ae8ad-21e6-4fde-afaa-549164cc1b1d"). InnerVolumeSpecName "kube-api-access-mmwjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:54:15.449650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.449627 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-podinfo" pod "186ae8ad-21e6-4fde-afaa-549164cc1b1d" (UID: "186ae8ad-21e6-4fde-afaa-549164cc1b1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:54:15.455432 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.452891 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-token" pod "186ae8ad-21e6-4fde-afaa-549164cc1b1d" (UID: "186ae8ad-21e6-4fde-afaa-549164cc1b1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:54:15.455432 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.453046 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-envoy" pod "186ae8ad-21e6-4fde-afaa-549164cc1b1d" (UID: "186ae8ad-21e6-4fde-afaa-549164cc1b1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:54:15.455703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.455601 5015 scope.go:122] "RemoveContainer" containerID="033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb" Nov 03 09:54:15.456039 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:54:15.456011 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb\": container with ID starting with 033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb not found: ID does not exist" containerID="033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb" Nov 03 09:54:15.456108 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.456052 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb"} err="failed to get container status \"033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb\": rpc error: code = NotFound desc = could not find container \"033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb\": container with ID starting with 033bc7f4b62a8f1e98f4fe1f628fd152f4042a13271f47e8d268c9c32ece7ecb not found: ID does not exist" Nov 03 09:54:15.456108 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.456077 5015 scope.go:122] "RemoveContainer" containerID="35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf" Nov 03 09:54:15.456402 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:54:15.456378 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf\": container with ID starting with 35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf not found: ID does not exist" containerID="35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf" Nov 03 09:54:15.456462 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.456413 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf"} err="failed to get container status \"35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf\": rpc error: code = NotFound desc = could not find container \"35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf\": container with ID starting with 35ef65f526825d21283eaf684ae22ba5353288ecf6555a226ef6408816b4e0cf not found: ID does not exist" Nov 03 09:54:15.545405 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.545370 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:15.545405 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.545408 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:15.545650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.545422 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:15.545650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.545448 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:15.545650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.545461 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:15.545650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.545519 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:15.545650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.545533 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:15.545650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.545547 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmwjg\" (UniqueName: \"kubernetes.io/projected/186ae8ad-21e6-4fde-afaa-549164cc1b1d-kube-api-access-mmwjg\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:15.545650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.545560 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186ae8ad-21e6-4fde-afaa-549164cc1b1d-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:15.852576 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.852538 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f"] Nov 03 09:54:15.861978 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:15.861950 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f8439f5b-openshift-default-5bd7cfbb4f-hwt6f"] Nov 03 09:54:16.913994 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:54:16.913958 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:54:17.205336 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:17.205259 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" path="/var/lib/kubelet/pods/186ae8ad-21e6-4fde-afaa-549164cc1b1d/volumes" Nov 03 09:54:25.408585 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:25.408550 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94"] Nov 03 09:54:25.522007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:25.521977 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7"] Nov 03 09:54:28.061577 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:28.061549 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:54:28.090219 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:28.090194 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:54:30.520995 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:30.520957 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" podUID="724c937b-6a9c-4c29-93d5-4a434f384a93" containerName="payload-processing" containerID="cri-o://3c838d7c6ecfd0cd7192e35c879c066c4634095a10ae366f270c9b85710db745" gracePeriod=25 Nov 03 09:54:30.522515 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:30.522463 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" podUID="8d86ce37-a32d-4d80-8331-f1f0b10c0048" containerName="payload-pre-processing" containerID="cri-o://cb15b8c76c488e41f89381d28bbc3c3b9202a541d6a8be887004632da07b259f" gracePeriod=25 Nov 03 09:54:34.400055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:34.400016 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" podUID="8d86ce37-a32d-4d80-8331-f1f0b10c0048" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:54:34.400495 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:34.400019 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" podUID="724c937b-6a9c-4c29-93d5-4a434f384a93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:54:44.399552 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:44.399511 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" podUID="724c937b-6a9c-4c29-93d5-4a434f384a93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:54:44.400083 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:44.399529 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" podUID="8d86ce37-a32d-4d80-8331-f1f0b10c0048" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:54:45.996985 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:45.996944 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5"] Nov 03 09:54:46.903305 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:54:46.903270 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:54:46.981131 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:46.981088 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv"] Nov 03 09:54:50.465427 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:50.465393 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4"] Nov 03 09:54:50.576611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:50.576582 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f"] Nov 03 09:54:50.757302 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:50.757169 5015 generic.go:356] "Generic (PLEG): container finished" podID="8d86ce37-a32d-4d80-8331-f1f0b10c0048" containerID="cb15b8c76c488e41f89381d28bbc3c3b9202a541d6a8be887004632da07b259f" exitCode=0 Nov 03 09:54:50.757302 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:50.757208 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" event={"ID":"8d86ce37-a32d-4d80-8331-f1f0b10c0048","Type":"ContainerDied","Data":"cb15b8c76c488e41f89381d28bbc3c3b9202a541d6a8be887004632da07b259f"} Nov 03 09:54:50.779130 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:50.779099 5015 generic.go:356] "Generic (PLEG): container finished" podID="724c937b-6a9c-4c29-93d5-4a434f384a93" containerID="3c838d7c6ecfd0cd7192e35c879c066c4634095a10ae366f270c9b85710db745" exitCode=0 Nov 03 09:54:50.779409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:50.779324 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" event={"ID":"724c937b-6a9c-4c29-93d5-4a434f384a93","Type":"ContainerDied","Data":"3c838d7c6ecfd0cd7192e35c879c066c4634095a10ae366f270c9b85710db745"} Nov 03 09:54:50.848339 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:50.848311 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 09:54:50.900702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:50.900668 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 09:54:51.027063 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.026966 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d86ce37-a32d-4d80-8331-f1f0b10c0048-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d86ce37-a32d-4d80-8331-f1f0b10c0048-plugins-config-volume\") pod \"8d86ce37-a32d-4d80-8331-f1f0b10c0048\" (UID: \"8d86ce37-a32d-4d80-8331-f1f0b10c0048\") " Nov 03 09:54:51.027749 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.027366 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d86ce37-a32d-4d80-8331-f1f0b10c0048-kube-api-access-xvhxk\" (UniqueName: \"kubernetes.io/projected/8d86ce37-a32d-4d80-8331-f1f0b10c0048-kube-api-access-xvhxk\") pod \"8d86ce37-a32d-4d80-8331-f1f0b10c0048\" (UID: \"8d86ce37-a32d-4d80-8331-f1f0b10c0048\") " Nov 03 09:54:51.027749 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.027408 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724c937b-6a9c-4c29-93d5-4a434f384a93-kube-api-access-tnfxp\" (UniqueName: \"kubernetes.io/projected/724c937b-6a9c-4c29-93d5-4a434f384a93-kube-api-access-tnfxp\") pod \"724c937b-6a9c-4c29-93d5-4a434f384a93\" (UID: \"724c937b-6a9c-4c29-93d5-4a434f384a93\") " Nov 03 09:54:51.027749 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.027513 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724c937b-6a9c-4c29-93d5-4a434f384a93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724c937b-6a9c-4c29-93d5-4a434f384a93-plugins-config-volume\") pod \"724c937b-6a9c-4c29-93d5-4a434f384a93\" (UID: \"724c937b-6a9c-4c29-93d5-4a434f384a93\") " Nov 03 09:54:51.030331 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.030285 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d86ce37-a32d-4d80-8331-f1f0b10c0048-plugins-config-volume" pod "8d86ce37-a32d-4d80-8331-f1f0b10c0048" (UID: "8d86ce37-a32d-4d80-8331-f1f0b10c0048"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:54:51.030558 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.030519 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724c937b-6a9c-4c29-93d5-4a434f384a93-plugins-config-volume" pod "724c937b-6a9c-4c29-93d5-4a434f384a93" (UID: "724c937b-6a9c-4c29-93d5-4a434f384a93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:54:51.031080 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.031049 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d86ce37-a32d-4d80-8331-f1f0b10c0048-kube-api-access-xvhxk" pod "8d86ce37-a32d-4d80-8331-f1f0b10c0048" (UID: "8d86ce37-a32d-4d80-8331-f1f0b10c0048"). InnerVolumeSpecName "kube-api-access-xvhxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:54:51.032561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.032536 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724c937b-6a9c-4c29-93d5-4a434f384a93-kube-api-access-tnfxp" pod "724c937b-6a9c-4c29-93d5-4a434f384a93" (UID: "724c937b-6a9c-4c29-93d5-4a434f384a93"). InnerVolumeSpecName "kube-api-access-tnfxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:54:51.130731 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.130698 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvhxk\" (UniqueName: \"kubernetes.io/projected/8d86ce37-a32d-4d80-8331-f1f0b10c0048-kube-api-access-xvhxk\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:51.130731 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.130726 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnfxp\" (UniqueName: \"kubernetes.io/projected/724c937b-6a9c-4c29-93d5-4a434f384a93-kube-api-access-tnfxp\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:51.130731 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.130735 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724c937b-6a9c-4c29-93d5-4a434f384a93-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:51.130967 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.130744 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d86ce37-a32d-4d80-8331-f1f0b10c0048-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:51.788852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.788822 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" Nov 03 09:54:51.788852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.788837 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7" event={"ID":"8d86ce37-a32d-4d80-8331-f1f0b10c0048","Type":"ContainerDied","Data":"e5a94f4b595657fee2dcc2db63bccf2e53bc1a8ef6490ad329d585d38e35e2d6"} Nov 03 09:54:51.789450 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.788909 5015 scope.go:122] "RemoveContainer" containerID="cb15b8c76c488e41f89381d28bbc3c3b9202a541d6a8be887004632da07b259f" Nov 03 09:54:51.790385 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.790360 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" event={"ID":"724c937b-6a9c-4c29-93d5-4a434f384a93","Type":"ContainerDied","Data":"e082603a12ffbb49168a6ac34277a795ea692dc11df721adca05ba71f9255781"} Nov 03 09:54:51.790557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.790535 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94" Nov 03 09:54:51.799432 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.799415 5015 scope.go:122] "RemoveContainer" containerID="3c838d7c6ecfd0cd7192e35c879c066c4634095a10ae366f270c9b85710db745" Nov 03 09:54:51.980800 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.980761 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94"] Nov 03 09:54:51.983471 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:51.983445 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18f9da-6c4f977b5-tlh94"] Nov 03 09:54:52.095145 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.095064 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7"] Nov 03 09:54:52.101632 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.101597 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-18f9da-fd7c59df9-7vlg7"] Nov 03 09:54:52.431849 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.431767 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24"] Nov 03 09:54:52.432007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.431975 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" podUID="48572303-fb5d-4f95-bf3b-7ba9e992d5e8" containerName="maas-api" containerID="cri-o://3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565" gracePeriod=30 Nov 03 09:54:52.746902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.746870 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:54:52.801028 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.800988 5015 generic.go:356] "Generic (PLEG): container finished" podID="48572303-fb5d-4f95-bf3b-7ba9e992d5e8" containerID="3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565" exitCode=0 Nov 03 09:54:52.801443 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.801034 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" Nov 03 09:54:52.801443 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.801067 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" event={"ID":"48572303-fb5d-4f95-bf3b-7ba9e992d5e8","Type":"ContainerDied","Data":"3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565"} Nov 03 09:54:52.801443 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.801158 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24" event={"ID":"48572303-fb5d-4f95-bf3b-7ba9e992d5e8","Type":"ContainerDied","Data":"0fd8f510fe84ddc8810e36e89319c11264d278e06b44a3936c7a211807320dba"} Nov 03 09:54:52.801443 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.801185 5015 scope.go:122] "RemoveContainer" containerID="3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565" Nov 03 09:54:52.811844 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.811816 5015 scope.go:122] "RemoveContainer" containerID="3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565" Nov 03 09:54:52.812114 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:54:52.812091 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565\": container with ID starting with 3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565 not found: ID does not exist" containerID="3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565" Nov 03 09:54:52.812193 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.812127 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565"} err="failed to get container status \"3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565\": rpc error: code = NotFound desc = could not find container \"3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565\": container with ID starting with 3fcddf2c6cc14fecc5cb48f82bf2da67a3515a3e7870237883a88546fa4b0565 not found: ID does not exist" Nov 03 09:54:52.851000 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.850971 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-kube-api-access-xmq9j\" (UniqueName: \"kubernetes.io/projected/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-kube-api-access-xmq9j\") pod \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " Nov 03 09:54:52.851188 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.851074 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-metrics-certs\") pod \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " Nov 03 09:54:52.851188 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.851099 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-openshift-service-ca\") pod \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " Nov 03 09:54:52.851312 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.851260 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-maas-api-tls\") pod \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\" (UID: \"48572303-fb5d-4f95-bf3b-7ba9e992d5e8\") " Nov 03 09:54:52.851619 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.851587 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-openshift-service-ca" pod "48572303-fb5d-4f95-bf3b-7ba9e992d5e8" (UID: "48572303-fb5d-4f95-bf3b-7ba9e992d5e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:54:52.851869 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.851848 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:52.853748 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.853726 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-maas-api-tls" pod "48572303-fb5d-4f95-bf3b-7ba9e992d5e8" (UID: "48572303-fb5d-4f95-bf3b-7ba9e992d5e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:54:52.854013 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.853985 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-metrics-certs" pod "48572303-fb5d-4f95-bf3b-7ba9e992d5e8" (UID: "48572303-fb5d-4f95-bf3b-7ba9e992d5e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:54:52.854013 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.853998 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-kube-api-access-xmq9j" pod "48572303-fb5d-4f95-bf3b-7ba9e992d5e8" (UID: "48572303-fb5d-4f95-bf3b-7ba9e992d5e8"). InnerVolumeSpecName "kube-api-access-xmq9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:54:52.953092 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.953054 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:52.953092 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.953095 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmq9j\" (UniqueName: \"kubernetes.io/projected/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-kube-api-access-xmq9j\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:52.953300 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:52.953110 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48572303-fb5d-4f95-bf3b-7ba9e992d5e8-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:53.204582 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:53.204548 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724c937b-6a9c-4c29-93d5-4a434f384a93" path="/var/lib/kubelet/pods/724c937b-6a9c-4c29-93d5-4a434f384a93/volumes" Nov 03 09:54:53.204997 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:53.204983 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d86ce37-a32d-4d80-8331-f1f0b10c0048" path="/var/lib/kubelet/pods/8d86ce37-a32d-4d80-8331-f1f0b10c0048/volumes" Nov 03 09:54:53.231179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:53.231150 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24"] Nov 03 09:54:53.238080 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:53.238052 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-433821-7986fc5d78-rnh24"] Nov 03 09:54:55.206718 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:55.206674 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48572303-fb5d-4f95-bf3b-7ba9e992d5e8" path="/var/lib/kubelet/pods/48572303-fb5d-4f95-bf3b-7ba9e992d5e8/volumes" Nov 03 09:54:55.466618 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:55.466503 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" podUID="4e5255ed-f323-4c2f-b431-14b4bcf956e4" containerName="payload-processing" containerID="cri-o://5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca" gracePeriod=25 Nov 03 09:54:55.577740 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:55.577689 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" podUID="d353fe68-ba7d-433c-b7b6-9dca79d7de10" containerName="payload-pre-processing" containerID="cri-o://d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f" gracePeriod=25 Nov 03 09:54:57.906047 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:57.906008 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" podUID="d353fe68-ba7d-433c-b7b6-9dca79d7de10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:54:57.906507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:54:57.906084 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" podUID="4e5255ed-f323-4c2f-b431-14b4bcf956e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:55:01.019538 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.019450 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" podUID="783d720f-d297-495b-b96a-299116fd1150" containerName="main" containerID="cri-o://05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc" gracePeriod=45 Nov 03 09:55:01.301613 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.301591 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:55:01.380329 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.380284 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-dshm\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-dshm\") pod \"783d720f-d297-495b-b96a-299116fd1150\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " Nov 03 09:55:01.380572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.380362 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/783d720f-d297-495b-b96a-299116fd1150-tls-certs\" (UniqueName: \"kubernetes.io/secret/783d720f-d297-495b-b96a-299116fd1150-tls-certs\") pod \"783d720f-d297-495b-b96a-299116fd1150\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " Nov 03 09:55:01.380572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.380384 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-home\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-home\") pod \"783d720f-d297-495b-b96a-299116fd1150\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " Nov 03 09:55:01.380572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.380425 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783d720f-d297-495b-b96a-299116fd1150-kube-api-access-g7tg5\" (UniqueName: \"kubernetes.io/projected/783d720f-d297-495b-b96a-299116fd1150-kube-api-access-g7tg5\") pod \"783d720f-d297-495b-b96a-299116fd1150\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " Nov 03 09:55:01.380572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.380492 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-tmp-dir\") pod \"783d720f-d297-495b-b96a-299116fd1150\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " Nov 03 09:55:01.380879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.380608 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-model-cache\") pod \"783d720f-d297-495b-b96a-299116fd1150\" (UID: \"783d720f-d297-495b-b96a-299116fd1150\") " Nov 03 09:55:01.380879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.380723 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-home" pod "783d720f-d297-495b-b96a-299116fd1150" (UID: "783d720f-d297-495b-b96a-299116fd1150"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:01.380879 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.380853 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-tmp-dir" pod "783d720f-d297-495b-b96a-299116fd1150" (UID: "783d720f-d297-495b-b96a-299116fd1150"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:01.381055 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.380920 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-model-cache" pod "783d720f-d297-495b-b96a-299116fd1150" (UID: "783d720f-d297-495b-b96a-299116fd1150"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:01.381368 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.381346 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:01.381440 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.381398 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:01.381440 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.381413 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:01.382916 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.382894 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/783d720f-d297-495b-b96a-299116fd1150-tls-certs" pod "783d720f-d297-495b-b96a-299116fd1150" (UID: "783d720f-d297-495b-b96a-299116fd1150"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:55:01.382916 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.382905 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783d720f-d297-495b-b96a-299116fd1150-kube-api-access-g7tg5" pod "783d720f-d297-495b-b96a-299116fd1150" (UID: "783d720f-d297-495b-b96a-299116fd1150"). InnerVolumeSpecName "kube-api-access-g7tg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:01.382916 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.382903 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-dshm" pod "783d720f-d297-495b-b96a-299116fd1150" (UID: "783d720f-d297-495b-b96a-299116fd1150"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:01.482974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.482929 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783d720f-d297-495b-b96a-299116fd1150-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:01.482974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.482967 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783d720f-d297-495b-b96a-299116fd1150-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:01.483201 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.482983 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7tg5\" (UniqueName: \"kubernetes.io/projected/783d720f-d297-495b-b96a-299116fd1150-kube-api-access-g7tg5\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:01.880858 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.880818 5015 generic.go:356] "Generic (PLEG): container finished" podID="783d720f-d297-495b-b96a-299116fd1150" containerID="05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc" exitCode=0 Nov 03 09:55:01.881065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.880869 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" Nov 03 09:55:01.881065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.880893 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" event={"ID":"783d720f-d297-495b-b96a-299116fd1150","Type":"ContainerDied","Data":"05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc"} Nov 03 09:55:01.881065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.880939 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5" event={"ID":"783d720f-d297-495b-b96a-299116fd1150","Type":"ContainerDied","Data":"5b8fe508702ece94d17097d0b19751260a5026ac4a2ba41928e1f07989511fea"} Nov 03 09:55:01.881065 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.880958 5015 scope.go:122] "RemoveContainer" containerID="05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc" Nov 03 09:55:01.893080 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.893058 5015 scope.go:122] "RemoveContainer" containerID="05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc" Nov 03 09:55:01.893388 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:55:01.893359 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc\": container with ID starting with 05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc not found: ID does not exist" containerID="05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc" Nov 03 09:55:01.893529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:01.893392 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc"} err="failed to get container status \"05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc\": rpc error: code = NotFound desc = could not find container \"05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc\": container with ID starting with 05d5970ae2b090505ec394d35742559f344b0bfb1dddba7fa0a39a3596f305dc not found: ID does not exist" Nov 03 09:55:02.002389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.002344 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" podUID="373ca1dd-aa36-4d69-af70-2970f27c1c97" containerName="main" containerID="cri-o://de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb" gracePeriod=45 Nov 03 09:55:02.050377 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.050340 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5"] Nov 03 09:55:02.057497 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.057452 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/rate-test-model-f9b517-kserve-6dddf5cc49-szls5"] Nov 03 09:55:02.280022 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.279992 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:55:02.392264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.392227 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373ca1dd-aa36-4d69-af70-2970f27c1c97-kube-api-access-6r92m\" (UniqueName: \"kubernetes.io/projected/373ca1dd-aa36-4d69-af70-2970f27c1c97-kube-api-access-6r92m\") pod \"373ca1dd-aa36-4d69-af70-2970f27c1c97\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " Nov 03 09:55:02.392467 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.392293 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373ca1dd-aa36-4d69-af70-2970f27c1c97-tls-certs\" (UniqueName: \"kubernetes.io/secret/373ca1dd-aa36-4d69-af70-2970f27c1c97-tls-certs\") pod \"373ca1dd-aa36-4d69-af70-2970f27c1c97\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " Nov 03 09:55:02.392467 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.392346 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-tmp-dir\") pod \"373ca1dd-aa36-4d69-af70-2970f27c1c97\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " Nov 03 09:55:02.392467 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.392420 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-model-cache\") pod \"373ca1dd-aa36-4d69-af70-2970f27c1c97\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " Nov 03 09:55:02.392676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.392543 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-home\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-home\") pod \"373ca1dd-aa36-4d69-af70-2970f27c1c97\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " Nov 03 09:55:02.392676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.392572 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-dshm\") pod \"373ca1dd-aa36-4d69-af70-2970f27c1c97\" (UID: \"373ca1dd-aa36-4d69-af70-2970f27c1c97\") " Nov 03 09:55:02.392676 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.392644 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-tmp-dir" pod "373ca1dd-aa36-4d69-af70-2970f27c1c97" (UID: "373ca1dd-aa36-4d69-af70-2970f27c1c97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:02.392833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.392722 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-model-cache" pod "373ca1dd-aa36-4d69-af70-2970f27c1c97" (UID: "373ca1dd-aa36-4d69-af70-2970f27c1c97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:02.392833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.392784 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-home" pod "373ca1dd-aa36-4d69-af70-2970f27c1c97" (UID: "373ca1dd-aa36-4d69-af70-2970f27c1c97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:02.393112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.393081 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:02.393112 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.393104 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:02.393234 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.393117 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:02.394962 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.394919 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373ca1dd-aa36-4d69-af70-2970f27c1c97-tls-certs" pod "373ca1dd-aa36-4d69-af70-2970f27c1c97" (UID: "373ca1dd-aa36-4d69-af70-2970f27c1c97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:55:02.396249 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.396221 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-dshm" pod "373ca1dd-aa36-4d69-af70-2970f27c1c97" (UID: "373ca1dd-aa36-4d69-af70-2970f27c1c97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:02.396380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.396287 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373ca1dd-aa36-4d69-af70-2970f27c1c97-kube-api-access-6r92m" pod "373ca1dd-aa36-4d69-af70-2970f27c1c97" (UID: "373ca1dd-aa36-4d69-af70-2970f27c1c97"). InnerVolumeSpecName "kube-api-access-6r92m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:02.494012 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.493972 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373ca1dd-aa36-4d69-af70-2970f27c1c97-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:02.494012 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.494006 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r92m\" (UniqueName: \"kubernetes.io/projected/373ca1dd-aa36-4d69-af70-2970f27c1c97-kube-api-access-6r92m\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:02.494230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.494020 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373ca1dd-aa36-4d69-af70-2970f27c1c97-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:02.893024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.892939 5015 generic.go:356] "Generic (PLEG): container finished" podID="373ca1dd-aa36-4d69-af70-2970f27c1c97" containerID="de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb" exitCode=0 Nov 03 09:55:02.893024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.892990 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" Nov 03 09:55:02.893233 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.892996 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" event={"ID":"373ca1dd-aa36-4d69-af70-2970f27c1c97","Type":"ContainerDied","Data":"de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb"} Nov 03 09:55:02.893233 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.893108 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv" event={"ID":"373ca1dd-aa36-4d69-af70-2970f27c1c97","Type":"ContainerDied","Data":"72e78d994d2a12785061bf913af9fba624c218bb27326df17e2506668213e63d"} Nov 03 09:55:02.893233 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.893140 5015 scope.go:122] "RemoveContainer" containerID="de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb" Nov 03 09:55:02.904809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.904788 5015 scope.go:122] "RemoveContainer" containerID="de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb" Nov 03 09:55:02.905067 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:55:02.905051 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb\": container with ID starting with de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb not found: ID does not exist" containerID="de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb" Nov 03 09:55:02.905113 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:02.905077 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb"} err="failed to get container status \"de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb\": rpc error: code = NotFound desc = could not find container \"de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb\": container with ID starting with de4b7a1fbfee9cafcab628bc2cf98d7f04605920440a299f3219b7ac04ed04eb not found: ID does not exist" Nov 03 09:55:03.066840 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:03.066795 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv"] Nov 03 09:55:03.115419 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:03.115383 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-506e6d/rate-test-model-506e6d-kserve-7c7cc6897c-skpbv"] Nov 03 09:55:03.204638 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:03.204565 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373ca1dd-aa36-4d69-af70-2970f27c1c97" path="/var/lib/kubelet/pods/373ca1dd-aa36-4d69-af70-2970f27c1c97/volumes" Nov 03 09:55:03.204995 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:03.204982 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783d720f-d297-495b-b96a-299116fd1150" path="/var/lib/kubelet/pods/783d720f-d297-495b-b96a-299116fd1150/volumes" Nov 03 09:55:07.906727 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:07.906643 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" podUID="4e5255ed-f323-4c2f-b431-14b4bcf956e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:55:07.906727 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:07.906680 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" podUID="d353fe68-ba7d-433c-b7b6-9dca79d7de10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:55:09.060534 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.060501 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq"] Nov 03 09:55:09.061060 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061047 5015 cpu_manager.go:414] "containerMap: removing container" podUID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" containerName="istio-proxy" Nov 03 09:55:09.061107 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061062 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" containerName="istio-proxy" Nov 03 09:55:09.061107 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061079 5015 cpu_manager.go:414] "containerMap: removing container" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" Nov 03 09:55:09.061107 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061084 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" Nov 03 09:55:09.061107 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061094 5015 cpu_manager.go:414] "containerMap: removing container" podUID="48572303-fb5d-4f95-bf3b-7ba9e992d5e8" containerName="maas-api" Nov 03 09:55:09.061107 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061100 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48572303-fb5d-4f95-bf3b-7ba9e992d5e8" containerName="maas-api" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061110 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8d86ce37-a32d-4d80-8331-f1f0b10c0048" containerName="payload-pre-processing" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061115 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d86ce37-a32d-4d80-8331-f1f0b10c0048" containerName="payload-pre-processing" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061124 5015 cpu_manager.go:414] "containerMap: removing container" podUID="373ca1dd-aa36-4d69-af70-2970f27c1c97" containerName="main" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061129 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373ca1dd-aa36-4d69-af70-2970f27c1c97" containerName="main" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061148 5015 cpu_manager.go:414] "containerMap: removing container" podUID="783d720f-d297-495b-b96a-299116fd1150" containerName="main" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061153 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783d720f-d297-495b-b96a-299116fd1150" containerName="main" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061161 5015 cpu_manager.go:414] "containerMap: removing container" podUID="724c937b-6a9c-4c29-93d5-4a434f384a93" containerName="payload-processing" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061165 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724c937b-6a9c-4c29-93d5-4a434f384a93" containerName="payload-processing" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061173 5015 cpu_manager.go:414] "containerMap: removing container" podUID="384591ef-c634-4c99-8e9b-4f85332e5802" containerName="main" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061177 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384591ef-c634-4c99-8e9b-4f85332e5802" containerName="main" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061264 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="783d720f-d297-495b-b96a-299116fd1150" containerName="main" Nov 03 09:55:09.061272 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061273 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="724c937b-6a9c-4c29-93d5-4a434f384a93" containerName="payload-processing" Nov 03 09:55:09.061693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061279 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="384591ef-c634-4c99-8e9b-4f85332e5802" containerName="main" Nov 03 09:55:09.061693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061288 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" Nov 03 09:55:09.061693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061293 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="228c6eb5-fcf2-422e-a4a6-19a6043bbe21" containerName="istio-proxy" Nov 03 09:55:09.061693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061300 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="48572303-fb5d-4f95-bf3b-7ba9e992d5e8" containerName="maas-api" Nov 03 09:55:09.061693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061306 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d86ce37-a32d-4d80-8331-f1f0b10c0048" containerName="payload-pre-processing" Nov 03 09:55:09.061693 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.061310 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="373ca1dd-aa36-4d69-af70-2970f27c1c97" containerName="main" Nov 03 09:55:09.064567 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.064551 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.119295 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.119271 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-18f9da-openshift-default-dockercfg-zmkbm\"" Nov 03 09:55:09.171367 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.171328 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.171646 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.171614 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48f8\" (UniqueName: \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-kube-api-access-w48f8\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.171794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.171723 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.171794 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.171772 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.171973 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.171840 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.171973 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.171922 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.172053 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.171978 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.172088 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.172064 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.172123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.172088 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.174126 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.174103 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq"] Nov 03 09:55:09.272791 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.272758 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.272987 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.272800 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48f8\" (UniqueName: \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-kube-api-access-w48f8\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.272987 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.272830 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.272987 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.272855 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.272987 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.272927 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.273245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.272991 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.273245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.273038 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.273245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.273144 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.273245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.273179 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.273245 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.273239 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.273596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.273344 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.273596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.273458 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.273596 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.273562 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.273952 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.273928 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.275469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.275445 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.275599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.275580 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.332177 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.332112 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48f8\" (UniqueName: \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-kube-api-access-w48f8\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.332177 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.332115 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.378706 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.378670 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:09.817703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.817660 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq"] Nov 03 09:55:09.854934 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:55:09.854902 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bae820b_3e21_4df3_be7f_7fc15df83f8c.slice/crio-f14a5bc93ed3baecfcbf280c0f7e72e8f1cc20653c806bfc13ecb8889dcc2efc WatchSource:0}: Error finding container f14a5bc93ed3baecfcbf280c0f7e72e8f1cc20653c806bfc13ecb8889dcc2efc: Status 404 returned error can't find the container with id f14a5bc93ed3baecfcbf280c0f7e72e8f1cc20653c806bfc13ecb8889dcc2efc Nov 03 09:55:09.857401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.857373 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:55:09.857520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.857437 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:55:09.857520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.857469 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:55:09.946187 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.946146 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" event={"ID":"1bae820b-3e21-4df3-be7f-7fc15df83f8c","Type":"ContainerStarted","Data":"21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09"} Nov 03 09:55:09.946362 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:09.946213 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" event={"ID":"1bae820b-3e21-4df3-be7f-7fc15df83f8c","Type":"ContainerStarted","Data":"f14a5bc93ed3baecfcbf280c0f7e72e8f1cc20653c806bfc13ecb8889dcc2efc"} Nov 03 09:55:10.067096 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:10.067040 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" podStartSLOduration=1.067026159 podStartE2EDuration="1.067026159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:55:10.064214658 +0000 UTC m=+4542.989156876" watchObservedRunningTime="2026-11-03 09:55:10.067026159 +0000 UTC m=+4542.991968379" Nov 03 09:55:10.378852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:10.378816 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:10.380767 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:10.380738 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 03 09:55:10.380895 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:10.380802 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 03 09:55:11.379864 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:11.379821 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 03 09:55:11.380408 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:11.379892 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 03 09:55:12.379814 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:12.379780 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 03 09:55:12.379998 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:12.379831 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 03 09:55:13.379594 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:13.379558 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 03 09:55:13.379786 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:13.379616 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 03 09:55:13.969721 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:13.969683 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9"] Nov 03 09:55:13.970664 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:13.970606 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" podUID="ed11b2ca-3e57-40d1-8eeb-20a31f9533c2" containerName="maas-api" containerID="cri-o://97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924" gracePeriod=30 Nov 03 09:55:14.295505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.295221 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 09:55:14.389715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.389254 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 03 09:55:14.389715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.389311 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 03 09:55:14.435499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.435169 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-kube-api-access-m4sct\" (UniqueName: \"kubernetes.io/projected/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-kube-api-access-m4sct\") pod \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " Nov 03 09:55:14.435499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.435229 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-metrics-certs\") pod \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " Nov 03 09:55:14.435499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.435296 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-openshift-service-ca\") pod \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " Nov 03 09:55:14.435499 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.435407 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-maas-api-tls\") pod \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\" (UID: \"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2\") " Nov 03 09:55:14.435904 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.435863 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-openshift-service-ca" pod "ed11b2ca-3e57-40d1-8eeb-20a31f9533c2" (UID: "ed11b2ca-3e57-40d1-8eeb-20a31f9533c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:55:14.436385 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.436199 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:14.467588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.462307 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-maas-api-tls" pod "ed11b2ca-3e57-40d1-8eeb-20a31f9533c2" (UID: "ed11b2ca-3e57-40d1-8eeb-20a31f9533c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:55:14.467588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.462936 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-kube-api-access-m4sct" pod "ed11b2ca-3e57-40d1-8eeb-20a31f9533c2" (UID: "ed11b2ca-3e57-40d1-8eeb-20a31f9533c2"). InnerVolumeSpecName "kube-api-access-m4sct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:14.468716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.468683 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-metrics-certs" pod "ed11b2ca-3e57-40d1-8eeb-20a31f9533c2" (UID: "ed11b2ca-3e57-40d1-8eeb-20a31f9533c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:55:14.567588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.549520 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:14.567588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.549563 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4sct\" (UniqueName: \"kubernetes.io/projected/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-kube-api-access-m4sct\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:14.567588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:14.549580 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:15.000689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.000649 5015 generic.go:356] "Generic (PLEG): container finished" podID="ed11b2ca-3e57-40d1-8eeb-20a31f9533c2" containerID="97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924" exitCode=0 Nov 03 09:55:15.001148 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.000701 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" Nov 03 09:55:15.001148 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.000722 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" event={"ID":"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2","Type":"ContainerDied","Data":"97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924"} Nov 03 09:55:15.001148 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.000779 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9" event={"ID":"ed11b2ca-3e57-40d1-8eeb-20a31f9533c2","Type":"ContainerDied","Data":"d614e62e473faf3a336424b4ed47cb179f2d4fc531393eb1df7321637d6b9e71"} Nov 03 09:55:15.001148 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.000801 5015 scope.go:122] "RemoveContainer" containerID="97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924" Nov 03 09:55:15.012164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.012148 5015 scope.go:122] "RemoveContainer" containerID="97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924" Nov 03 09:55:15.012411 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:55:15.012390 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924\": container with ID starting with 97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924 not found: ID does not exist" containerID="97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924" Nov 03 09:55:15.012458 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.012421 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924"} err="failed to get container status \"97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924\": rpc error: code = NotFound desc = could not find container \"97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924\": container with ID starting with 97ea1ba2c026c65e02cc185eaf49f5ee70e9906820e9e0c8b24234eaaeef0924 not found: ID does not exist" Nov 03 09:55:15.174100 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.174068 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9"] Nov 03 09:55:15.224195 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.224168 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-18f9da-6fc87cd7b5-4zvx9"] Nov 03 09:55:15.379977 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.379891 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 03 09:55:15.379977 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.379958 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 03 09:55:15.809023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.808125 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:55:15.867571 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.867548 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:55:15.978426 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.978392 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5255ed-f323-4c2f-b431-14b4bcf956e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5255ed-f323-4c2f-b431-14b4bcf956e4-plugins-config-volume\") pod \"4e5255ed-f323-4c2f-b431-14b4bcf956e4\" (UID: \"4e5255ed-f323-4c2f-b431-14b4bcf956e4\") " Nov 03 09:55:15.978672 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.978523 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d353fe68-ba7d-433c-b7b6-9dca79d7de10-kube-api-access-2qc9c\" (UniqueName: \"kubernetes.io/projected/d353fe68-ba7d-433c-b7b6-9dca79d7de10-kube-api-access-2qc9c\") pod \"d353fe68-ba7d-433c-b7b6-9dca79d7de10\" (UID: \"d353fe68-ba7d-433c-b7b6-9dca79d7de10\") " Nov 03 09:55:15.978672 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.978596 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5255ed-f323-4c2f-b431-14b4bcf956e4-kube-api-access-shpwr\" (UniqueName: \"kubernetes.io/projected/4e5255ed-f323-4c2f-b431-14b4bcf956e4-kube-api-access-shpwr\") pod \"4e5255ed-f323-4c2f-b431-14b4bcf956e4\" (UID: \"4e5255ed-f323-4c2f-b431-14b4bcf956e4\") " Nov 03 09:55:15.978672 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.978640 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d353fe68-ba7d-433c-b7b6-9dca79d7de10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353fe68-ba7d-433c-b7b6-9dca79d7de10-plugins-config-volume\") pod \"d353fe68-ba7d-433c-b7b6-9dca79d7de10\" (UID: \"d353fe68-ba7d-433c-b7b6-9dca79d7de10\") " Nov 03 09:55:15.979041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.978881 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5255ed-f323-4c2f-b431-14b4bcf956e4-plugins-config-volume" pod "4e5255ed-f323-4c2f-b431-14b4bcf956e4" (UID: "4e5255ed-f323-4c2f-b431-14b4bcf956e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:55:15.979185 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.979082 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d353fe68-ba7d-433c-b7b6-9dca79d7de10-plugins-config-volume" pod "d353fe68-ba7d-433c-b7b6-9dca79d7de10" (UID: "d353fe68-ba7d-433c-b7b6-9dca79d7de10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:55:15.979259 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.979232 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353fe68-ba7d-433c-b7b6-9dca79d7de10-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:15.979324 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.979257 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5255ed-f323-4c2f-b431-14b4bcf956e4-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:15.981074 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.981052 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5255ed-f323-4c2f-b431-14b4bcf956e4-kube-api-access-shpwr" pod "4e5255ed-f323-4c2f-b431-14b4bcf956e4" (UID: "4e5255ed-f323-4c2f-b431-14b4bcf956e4"). InnerVolumeSpecName "kube-api-access-shpwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:15.981207 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:15.981114 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d353fe68-ba7d-433c-b7b6-9dca79d7de10-kube-api-access-2qc9c" pod "d353fe68-ba7d-433c-b7b6-9dca79d7de10" (UID: "d353fe68-ba7d-433c-b7b6-9dca79d7de10"). InnerVolumeSpecName "kube-api-access-2qc9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:16.010074 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.010038 5015 generic.go:356] "Generic (PLEG): container finished" podID="4e5255ed-f323-4c2f-b431-14b4bcf956e4" containerID="5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca" exitCode=0 Nov 03 09:55:16.010536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.010084 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" Nov 03 09:55:16.010536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.010116 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" event={"ID":"4e5255ed-f323-4c2f-b431-14b4bcf956e4","Type":"ContainerDied","Data":"5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca"} Nov 03 09:55:16.010536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.010178 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4" event={"ID":"4e5255ed-f323-4c2f-b431-14b4bcf956e4","Type":"ContainerDied","Data":"dad44f8e7000081d70aebac9db4f198d88b7f69634bf003ac2a638d6995e72f0"} Nov 03 09:55:16.010536 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.010201 5015 scope.go:122] "RemoveContainer" containerID="5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca" Nov 03 09:55:16.012396 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.012370 5015 generic.go:356] "Generic (PLEG): container finished" podID="d353fe68-ba7d-433c-b7b6-9dca79d7de10" containerID="d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f" exitCode=0 Nov 03 09:55:16.012511 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.012452 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" event={"ID":"d353fe68-ba7d-433c-b7b6-9dca79d7de10","Type":"ContainerDied","Data":"d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f"} Nov 03 09:55:16.012571 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.012512 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" event={"ID":"d353fe68-ba7d-433c-b7b6-9dca79d7de10","Type":"ContainerDied","Data":"d1ab493ea8c1fefd288e422279665f2b3f4366c8d5b62a402388b8c217fb9ea4"} Nov 03 09:55:16.012636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.012591 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f" Nov 03 09:55:16.022537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.022511 5015 scope.go:122] "RemoveContainer" containerID="5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca" Nov 03 09:55:16.023012 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:55:16.022982 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca\": container with ID starting with 5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca not found: ID does not exist" containerID="5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca" Nov 03 09:55:16.023101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.023021 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca"} err="failed to get container status \"5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca\": rpc error: code = NotFound desc = could not find container \"5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca\": container with ID starting with 5dff524b92651450c1def81ca6386ea918c3f135bc87c4c7c88a06eab0e36dca not found: ID does not exist" Nov 03 09:55:16.023101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.023045 5015 scope.go:122] "RemoveContainer" containerID="d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f" Nov 03 09:55:16.034538 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.034519 5015 scope.go:122] "RemoveContainer" containerID="d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f" Nov 03 09:55:16.034847 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:55:16.034821 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f\": container with ID starting with d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f not found: ID does not exist" containerID="d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f" Nov 03 09:55:16.034946 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.034854 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f"} err="failed to get container status \"d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f\": rpc error: code = NotFound desc = could not find container \"d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f\": container with ID starting with d6724c3a1811b56be26cd66518d08f8960aeed63852625d8878f2b4309069f2f not found: ID does not exist" Nov 03 09:55:16.080531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.080457 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qc9c\" (UniqueName: \"kubernetes.io/projected/d353fe68-ba7d-433c-b7b6-9dca79d7de10-kube-api-access-2qc9c\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:16.080531 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.080506 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shpwr\" (UniqueName: \"kubernetes.io/projected/4e5255ed-f323-4c2f-b431-14b4bcf956e4-kube-api-access-shpwr\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:16.160224 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.160184 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4"] Nov 03 09:55:16.167360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.167327 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-433821-5f77c5479f-w8sz4"] Nov 03 09:55:16.283883 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.283769 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f"] Nov 03 09:55:16.289696 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.289662 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-433821-7bd4fd9fd7-28v6f"] Nov 03 09:55:16.379992 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.379955 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 03 09:55:16.380156 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:16.380024 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 03 09:55:16.908165 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:55:16.908130 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:55:17.207974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:17.207888 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5255ed-f323-4c2f-b431-14b4bcf956e4" path="/var/lib/kubelet/pods/4e5255ed-f323-4c2f-b431-14b4bcf956e4/volumes" Nov 03 09:55:17.208678 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:17.208654 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d353fe68-ba7d-433c-b7b6-9dca79d7de10" path="/var/lib/kubelet/pods/d353fe68-ba7d-433c-b7b6-9dca79d7de10/volumes" Nov 03 09:55:17.209108 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:17.209091 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed11b2ca-3e57-40d1-8eeb-20a31f9533c2" path="/var/lib/kubelet/pods/ed11b2ca-3e57-40d1-8eeb-20a31f9533c2/volumes" Nov 03 09:55:17.937520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:17.934094 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq"] Nov 03 09:55:17.937520 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:17.934427 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" containerID="cri-o://21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09" gracePeriod=30 Nov 03 09:55:18.380148 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:18.380105 5015 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:55:18.380630 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:18.380179 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:55:23.876414 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.876382 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:23.985507 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985440 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-credential-socket\") pod \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " Nov 03 09:55:23.985688 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985515 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-token\" (UniqueName: \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-token\") pod \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " Nov 03 09:55:23.985688 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985579 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-socket\") pod \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " Nov 03 09:55:23.985688 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985644 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istiod-ca-cert\") pod \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " Nov 03 09:55:23.985688 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985680 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-certs\") pod \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " Nov 03 09:55:23.985908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985738 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-podinfo\") pod \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " Nov 03 09:55:23.985908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985773 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-credential-socket" pod "1bae820b-3e21-4df3-be7f-7fc15df83f8c" (UID: "1bae820b-3e21-4df3-be7f-7fc15df83f8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:23.985908 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985888 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-envoy\") pod \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " Nov 03 09:55:23.986081 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985924 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-data\") pod \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " Nov 03 09:55:23.986081 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985939 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-socket" pod "1bae820b-3e21-4df3-be7f-7fc15df83f8c" (UID: "1bae820b-3e21-4df3-be7f-7fc15df83f8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:23.986081 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985954 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-kube-api-access-w48f8\" (UniqueName: \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-kube-api-access-w48f8\") pod \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\" (UID: \"1bae820b-3e21-4df3-be7f-7fc15df83f8c\") " Nov 03 09:55:23.986081 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.985981 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-certs" pod "1bae820b-3e21-4df3-be7f-7fc15df83f8c" (UID: "1bae820b-3e21-4df3-be7f-7fc15df83f8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:23.986081 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.986040 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istiod-ca-cert" pod "1bae820b-3e21-4df3-be7f-7fc15df83f8c" (UID: "1bae820b-3e21-4df3-be7f-7fc15df83f8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:55:23.986309 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.986189 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-data" pod "1bae820b-3e21-4df3-be7f-7fc15df83f8c" (UID: "1bae820b-3e21-4df3-be7f-7fc15df83f8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:23.986429 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.986410 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.986504 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.986437 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.986569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.986518 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.986569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.986536 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.986569 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.986551 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.988168 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.988146 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-envoy" pod "1bae820b-3e21-4df3-be7f-7fc15df83f8c" (UID: "1bae820b-3e21-4df3-be7f-7fc15df83f8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:23.988292 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.988243 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-podinfo" pod "1bae820b-3e21-4df3-be7f-7fc15df83f8c" (UID: "1bae820b-3e21-4df3-be7f-7fc15df83f8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:55:23.988632 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.988612 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-token" pod "1bae820b-3e21-4df3-be7f-7fc15df83f8c" (UID: "1bae820b-3e21-4df3-be7f-7fc15df83f8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:23.988682 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:23.988638 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-kube-api-access-w48f8" pod "1bae820b-3e21-4df3-be7f-7fc15df83f8c" (UID: "1bae820b-3e21-4df3-be7f-7fc15df83f8c"). InnerVolumeSpecName "kube-api-access-w48f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:24.093976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.087348 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:24.093976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.087387 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w48f8\" (UniqueName: \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-kube-api-access-w48f8\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:24.093976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.087404 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:24.093976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.087416 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bae820b-3e21-4df3-be7f-7fc15df83f8c-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:24.112503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.112348 5015 generic.go:356] "Generic (PLEG): container finished" podID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerID="21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09" exitCode=0 Nov 03 09:55:24.112503 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.112432 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" Nov 03 09:55:24.113454 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.113004 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" event={"ID":"1bae820b-3e21-4df3-be7f-7fc15df83f8c","Type":"ContainerDied","Data":"21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09"} Nov 03 09:55:24.113454 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.113073 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq" event={"ID":"1bae820b-3e21-4df3-be7f-7fc15df83f8c","Type":"ContainerDied","Data":"f14a5bc93ed3baecfcbf280c0f7e72e8f1cc20653c806bfc13ecb8889dcc2efc"} Nov 03 09:55:24.113454 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.113105 5015 scope.go:122] "RemoveContainer" containerID="21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09" Nov 03 09:55:24.133796 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.133771 5015 scope.go:122] "RemoveContainer" containerID="21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09" Nov 03 09:55:24.134139 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:55:24.134115 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09\": container with ID starting with 21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09 not found: ID does not exist" containerID="21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09" Nov 03 09:55:24.134233 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.134150 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09"} err="failed to get container status \"21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09\": rpc error: code = NotFound desc = could not find container \"21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09\": container with ID starting with 21ca9bf711eea65ce41690d37f7b46a0ce3fa0d809ec3300725d7dbdc3f8bc09 not found: ID does not exist" Nov 03 09:55:24.282762 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.282727 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq"] Nov 03 09:55:24.287105 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:24.287080 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-j7jsq"] Nov 03 09:55:25.204989 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:25.204953 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" path="/var/lib/kubelet/pods/1bae820b-3e21-4df3-be7f-7fc15df83f8c/volumes" Nov 03 09:55:29.125898 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:29.125866 5015 scope.go:122] "RemoveContainer" containerID="ccfcf9681929097698e858fa30edc139bd4915e85b46e26a5c56e29b2d02e6f0" Nov 03 09:55:46.902205 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:55:46.902165 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:55:51.729398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:51.729361 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh"] Nov 03 09:55:51.729893 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:51.729602 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" containerID="cri-o://c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091" gracePeriod=30 Nov 03 09:55:52.744238 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:52.744192 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7"] Nov 03 09:55:52.744873 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:52.744444 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" podUID="a39c892d-ff3a-43d4-bcad-c77218158498" containerName="maas-api" containerID="cri-o://0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f" gracePeriod=30 Nov 03 09:55:53.025069 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.025038 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:55:53.119858 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.119827 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39c892d-ff3a-43d4-bcad-c77218158498-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39c892d-ff3a-43d4-bcad-c77218158498-openshift-service-ca\") pod \"a39c892d-ff3a-43d4-bcad-c77218158498\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " Nov 03 09:55:53.120064 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.119948 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-maas-api-tls\") pod \"a39c892d-ff3a-43d4-bcad-c77218158498\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " Nov 03 09:55:53.120064 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.119991 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39c892d-ff3a-43d4-bcad-c77218158498-kube-api-access-fh8lw\" (UniqueName: \"kubernetes.io/projected/a39c892d-ff3a-43d4-bcad-c77218158498-kube-api-access-fh8lw\") pod \"a39c892d-ff3a-43d4-bcad-c77218158498\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " Nov 03 09:55:53.120192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.120100 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-metrics-certs\") pod \"a39c892d-ff3a-43d4-bcad-c77218158498\" (UID: \"a39c892d-ff3a-43d4-bcad-c77218158498\") " Nov 03 09:55:53.120254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.120201 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39c892d-ff3a-43d4-bcad-c77218158498-openshift-service-ca" pod "a39c892d-ff3a-43d4-bcad-c77218158498" (UID: "a39c892d-ff3a-43d4-bcad-c77218158498"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:55:53.120574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.120555 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39c892d-ff3a-43d4-bcad-c77218158498-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:53.122355 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.122328 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39c892d-ff3a-43d4-bcad-c77218158498-kube-api-access-fh8lw" pod "a39c892d-ff3a-43d4-bcad-c77218158498" (UID: "a39c892d-ff3a-43d4-bcad-c77218158498"). InnerVolumeSpecName "kube-api-access-fh8lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:53.122470 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.122328 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-metrics-certs" pod "a39c892d-ff3a-43d4-bcad-c77218158498" (UID: "a39c892d-ff3a-43d4-bcad-c77218158498"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:55:53.122470 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.122373 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-maas-api-tls" pod "a39c892d-ff3a-43d4-bcad-c77218158498" (UID: "a39c892d-ff3a-43d4-bcad-c77218158498"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:55:53.221404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.221375 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:53.221404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.221403 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39c892d-ff3a-43d4-bcad-c77218158498-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:53.221602 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.221414 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh8lw\" (UniqueName: \"kubernetes.io/projected/a39c892d-ff3a-43d4-bcad-c77218158498-kube-api-access-fh8lw\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:53.320056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.319975 5015 generic.go:356] "Generic (PLEG): container finished" podID="a39c892d-ff3a-43d4-bcad-c77218158498" containerID="0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f" exitCode=0 Nov 03 09:55:53.320056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.320017 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" Nov 03 09:55:53.320056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.320040 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" event={"ID":"a39c892d-ff3a-43d4-bcad-c77218158498","Type":"ContainerDied","Data":"0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f"} Nov 03 09:55:53.320402 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.320080 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7" event={"ID":"a39c892d-ff3a-43d4-bcad-c77218158498","Type":"ContainerDied","Data":"930d867497ca88a8a5c32ba334a9cb806761fc689eae6c6ee2bfae1dc3a39da0"} Nov 03 09:55:53.320402 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.320101 5015 scope.go:122] "RemoveContainer" containerID="0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f" Nov 03 09:55:53.336647 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.336620 5015 scope.go:122] "RemoveContainer" containerID="0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f" Nov 03 09:55:53.336990 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:55:53.336960 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f\": container with ID starting with 0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f not found: ID does not exist" containerID="0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f" Nov 03 09:55:53.337063 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.336997 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f"} err="failed to get container status \"0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f\": rpc error: code = NotFound desc = could not find container \"0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f\": container with ID starting with 0d08a2b2d19c94be0f4ba7b6872f92a7f6a466f8ff73c01d58002a2af52e0e1f not found: ID does not exist" Nov 03 09:55:53.489611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.489574 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7"] Nov 03 09:55:53.494868 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:53.494837 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-f9b517-5f5fbfdf67-rrph7"] Nov 03 09:55:55.204773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:55.204744 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39c892d-ff3a-43d4-bcad-c77218158498" path="/var/lib/kubelet/pods/a39c892d-ff3a-43d4-bcad-c77218158498/volumes" Nov 03 09:55:56.993502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:56.993401 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:55:57.071972 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.071945 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istiod-ca-cert\") pod \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " Nov 03 09:55:57.072152 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072029 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-podinfo\") pod \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " Nov 03 09:55:57.072152 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072066 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-token\" (UniqueName: \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-token\") pod \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " Nov 03 09:55:57.072152 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072088 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-certs\") pod \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " Nov 03 09:55:57.072152 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072120 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-socket\") pod \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " Nov 03 09:55:57.072351 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072161 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-envoy\") pod \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " Nov 03 09:55:57.072351 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072250 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-kube-api-access-w9rgt\" (UniqueName: \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-kube-api-access-w9rgt\") pod \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " Nov 03 09:55:57.072466 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072368 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istiod-ca-cert" pod "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" (UID: "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:55:57.072571 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072505 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-socket" pod "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" (UID: "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:57.072571 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072551 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-certs" pod "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" (UID: "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:57.072677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072625 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-credential-socket\") pod \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " Nov 03 09:55:57.072677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072662 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-data\") pod \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\" (UID: \"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d\") " Nov 03 09:55:57.072884 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072861 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-credential-socket" pod "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" (UID: "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:57.072979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.072955 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-data" pod "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" (UID: "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:57.073262 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.073246 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:57.073349 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.073267 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:57.073349 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.073284 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:57.073349 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.073296 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:57.073349 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.073309 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:57.074984 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.074958 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-podinfo" pod "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" (UID: "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:55:57.075100 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.075063 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-kube-api-access-w9rgt" pod "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" (UID: "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d"). InnerVolumeSpecName "kube-api-access-w9rgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:57.075165 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.075141 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-token" pod "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" (UID: "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:57.075331 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.075317 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-envoy" pod "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" (UID: "448ff7d3-7b6e-49e6-9e3e-5093e589dc4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:57.174935 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.174901 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:57.174935 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.174936 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:57.175779 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.174951 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:57.175779 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.174968 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rgt\" (UniqueName: \"kubernetes.io/projected/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d-kube-api-access-w9rgt\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:57.374874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.374788 5015 generic.go:356] "Generic (PLEG): container finished" podID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerID="c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091" exitCode=0 Nov 03 09:55:57.374874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.374838 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" Nov 03 09:55:57.375084 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.374870 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" event={"ID":"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d","Type":"ContainerDied","Data":"c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091"} Nov 03 09:55:57.375084 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.374932 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh" event={"ID":"448ff7d3-7b6e-49e6-9e3e-5093e589dc4d","Type":"ContainerDied","Data":"75a437dfc9d08c98eab44502a1e020921cfe8ccf3e96abb6df17c583620f9887"} Nov 03 09:55:57.375084 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.374963 5015 scope.go:122] "RemoveContainer" containerID="c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091" Nov 03 09:55:57.385447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.385429 5015 scope.go:122] "RemoveContainer" containerID="ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6" Nov 03 09:55:57.396583 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.396556 5015 scope.go:122] "RemoveContainer" containerID="c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091" Nov 03 09:55:57.396860 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:55:57.396840 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091\": container with ID starting with c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091 not found: ID does not exist" containerID="c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091" Nov 03 09:55:57.396921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.396870 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091"} err="failed to get container status \"c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091\": rpc error: code = NotFound desc = could not find container \"c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091\": container with ID starting with c8fc5a62a9636d24ed4ca68419d6887d3aff0319b21ba951e1fddc83d7024091 not found: ID does not exist" Nov 03 09:55:57.396921 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.396887 5015 scope.go:122] "RemoveContainer" containerID="ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6" Nov 03 09:55:57.397121 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:55:57.397104 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6\": container with ID starting with ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6 not found: ID does not exist" containerID="ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6" Nov 03 09:55:57.397163 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.397127 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6"} err="failed to get container status \"ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6\": rpc error: code = NotFound desc = could not find container \"ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6\": container with ID starting with ed1f41f23133705e3eb96ef95b73fea22163944f451b7d420357862b3a10b1e6 not found: ID does not exist" Nov 03 09:55:57.545838 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.545801 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh"] Nov 03 09:55:57.596344 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:57.596305 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-433821-openshift-default-58f9cc5cbf-m67gh"] Nov 03 09:55:59.205518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:55:59.205458 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" path="/var/lib/kubelet/pods/448ff7d3-7b6e-49e6-9e3e-5093e589dc4d/volumes" Nov 03 09:56:10.679965 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:10.679933 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8"] Nov 03 09:56:10.788134 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:10.788093 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7"] Nov 03 09:56:15.681190 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:15.681134 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" podUID="4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb" containerName="payload-processing" containerID="cri-o://8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db" gracePeriod=25 Nov 03 09:56:15.789090 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:15.789048 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" podUID="20ad04c6-7102-4548-b311-81690a614cc9" containerName="payload-pre-processing" containerID="cri-o://8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072" gracePeriod=25 Nov 03 09:56:16.902144 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:16.902103 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:56:24.026116 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:24.026072 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" podUID="4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:56:24.026669 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:24.026077 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" podUID="20ad04c6-7102-4548-b311-81690a614cc9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:56:24.263905 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:24.263863 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w"] Nov 03 09:56:24.373763 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:24.373686 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb"] Nov 03 09:56:29.264403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:29.264359 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" podUID="eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1" containerName="payload-processing" containerID="cri-o://3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82" gracePeriod=25 Nov 03 09:56:29.374171 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:29.374131 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" podUID="1cc5a185-5f49-47f1-a634-65346de5e8b8" containerName="payload-pre-processing" containerID="cri-o://4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7" gracePeriod=25 Nov 03 09:56:31.134249 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:31.134206 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" podUID="1cc5a185-5f49-47f1-a634-65346de5e8b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:56:31.136720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:31.134342 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" podUID="eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:56:32.132031 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.131990 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb"] Nov 03 09:56:32.132283 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.132255 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" podUID="850f7210-fa1a-409c-b999-c76ffb9de8ad" containerName="maas-api" containerID="cri-o://d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be" gracePeriod=30 Nov 03 09:56:32.398634 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.398561 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:56:32.472537 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.472503 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850f7210-fa1a-409c-b999-c76ffb9de8ad-kube-api-access-gw284\" (UniqueName: \"kubernetes.io/projected/850f7210-fa1a-409c-b999-c76ffb9de8ad-kube-api-access-gw284\") pod \"850f7210-fa1a-409c-b999-c76ffb9de8ad\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " Nov 03 09:56:32.472751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.472632 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-metrics-certs\") pod \"850f7210-fa1a-409c-b999-c76ffb9de8ad\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " Nov 03 09:56:32.472751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.472675 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850f7210-fa1a-409c-b999-c76ffb9de8ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850f7210-fa1a-409c-b999-c76ffb9de8ad-openshift-service-ca\") pod \"850f7210-fa1a-409c-b999-c76ffb9de8ad\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " Nov 03 09:56:32.472877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.472752 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-maas-api-tls\") pod \"850f7210-fa1a-409c-b999-c76ffb9de8ad\" (UID: \"850f7210-fa1a-409c-b999-c76ffb9de8ad\") " Nov 03 09:56:32.473178 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.473129 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850f7210-fa1a-409c-b999-c76ffb9de8ad-openshift-service-ca" pod "850f7210-fa1a-409c-b999-c76ffb9de8ad" (UID: "850f7210-fa1a-409c-b999-c76ffb9de8ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:32.475139 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.475109 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-maas-api-tls" pod "850f7210-fa1a-409c-b999-c76ffb9de8ad" (UID: "850f7210-fa1a-409c-b999-c76ffb9de8ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:56:32.475284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.475109 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850f7210-fa1a-409c-b999-c76ffb9de8ad-kube-api-access-gw284" pod "850f7210-fa1a-409c-b999-c76ffb9de8ad" (UID: "850f7210-fa1a-409c-b999-c76ffb9de8ad"). InnerVolumeSpecName "kube-api-access-gw284". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:32.475488 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.475454 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-metrics-certs" pod "850f7210-fa1a-409c-b999-c76ffb9de8ad" (UID: "850f7210-fa1a-409c-b999-c76ffb9de8ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:56:32.578247 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.574936 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:32.578247 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.574980 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw284\" (UniqueName: \"kubernetes.io/projected/850f7210-fa1a-409c-b999-c76ffb9de8ad-kube-api-access-gw284\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:32.578247 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.575014 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850f7210-fa1a-409c-b999-c76ffb9de8ad-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:32.578247 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.575031 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850f7210-fa1a-409c-b999-c76ffb9de8ad-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:32.639377 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.636809 5015 generic.go:356] "Generic (PLEG): container finished" podID="850f7210-fa1a-409c-b999-c76ffb9de8ad" containerID="d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be" exitCode=0 Nov 03 09:56:32.641050 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.641014 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" Nov 03 09:56:32.641590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.641550 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" event={"ID":"850f7210-fa1a-409c-b999-c76ffb9de8ad","Type":"ContainerDied","Data":"d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be"} Nov 03 09:56:32.641736 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.641652 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb" event={"ID":"850f7210-fa1a-409c-b999-c76ffb9de8ad","Type":"ContainerDied","Data":"aa5c65578df03b5952ef1f5899bfd8ae0e5ad8fa8170355f6bfa8547e0003b95"} Nov 03 09:56:32.641736 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.641723 5015 scope.go:122] "RemoveContainer" containerID="d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be" Nov 03 09:56:32.667062 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.667027 5015 scope.go:122] "RemoveContainer" containerID="d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be" Nov 03 09:56:32.667461 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:32.667439 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be\": container with ID starting with d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be not found: ID does not exist" containerID="d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be" Nov 03 09:56:32.667545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.667488 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be"} err="failed to get container status \"d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be\": rpc error: code = NotFound desc = could not find container \"d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be\": container with ID starting with d6020733107c9d4102f0bf0c8b0520d582c309a3596a3b43fe815fb92c8339be not found: ID does not exist" Nov 03 09:56:32.810312 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.810277 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb"] Nov 03 09:56:32.815721 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:32.815693 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-627e5f-6cc5866575-8ksfb"] Nov 03 09:56:33.204294 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:33.204263 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850f7210-fa1a-409c-b999-c76ffb9de8ad" path="/var/lib/kubelet/pods/850f7210-fa1a-409c-b999-c76ffb9de8ad/volumes" Nov 03 09:56:34.025628 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:34.025588 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" podUID="20ad04c6-7102-4548-b311-81690a614cc9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:56:34.026047 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:34.025682 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" podUID="4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:56:35.954953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:35.954872 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:56:36.033117 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.033085 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-plugins-config-volume\") pod \"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb\" (UID: \"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb\") " Nov 03 09:56:36.033117 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.033124 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-kube-api-access-h45pv\" (UniqueName: \"kubernetes.io/projected/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-kube-api-access-h45pv\") pod \"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb\" (UID: \"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb\") " Nov 03 09:56:36.033605 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.033578 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-plugins-config-volume" pod "4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb" (UID: "4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:36.035937 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.035899 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-kube-api-access-h45pv" pod "4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb" (UID: "4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb"). InnerVolumeSpecName "kube-api-access-h45pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:36.055343 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.055314 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:56:36.134922 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.134830 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ad04c6-7102-4548-b311-81690a614cc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ad04c6-7102-4548-b311-81690a614cc9-plugins-config-volume\") pod \"20ad04c6-7102-4548-b311-81690a614cc9\" (UID: \"20ad04c6-7102-4548-b311-81690a614cc9\") " Nov 03 09:56:36.135100 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.134982 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ad04c6-7102-4548-b311-81690a614cc9-kube-api-access-q4x9f\" (UniqueName: \"kubernetes.io/projected/20ad04c6-7102-4548-b311-81690a614cc9-kube-api-access-q4x9f\") pod \"20ad04c6-7102-4548-b311-81690a614cc9\" (UID: \"20ad04c6-7102-4548-b311-81690a614cc9\") " Nov 03 09:56:36.135174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.135147 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ad04c6-7102-4548-b311-81690a614cc9-plugins-config-volume" pod "20ad04c6-7102-4548-b311-81690a614cc9" (UID: "20ad04c6-7102-4548-b311-81690a614cc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:36.135465 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.135445 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ad04c6-7102-4548-b311-81690a614cc9-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:36.135465 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.135465 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:36.135671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.135513 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h45pv\" (UniqueName: \"kubernetes.io/projected/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb-kube-api-access-h45pv\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:36.137895 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.137865 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ad04c6-7102-4548-b311-81690a614cc9-kube-api-access-q4x9f" pod "20ad04c6-7102-4548-b311-81690a614cc9" (UID: "20ad04c6-7102-4548-b311-81690a614cc9"). InnerVolumeSpecName "kube-api-access-q4x9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:36.236377 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.236342 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4x9f\" (UniqueName: \"kubernetes.io/projected/20ad04c6-7102-4548-b311-81690a614cc9-kube-api-access-q4x9f\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:36.667674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.667570 5015 generic.go:356] "Generic (PLEG): container finished" podID="4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb" containerID="8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db" exitCode=0 Nov 03 09:56:36.667674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.667612 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" event={"ID":"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb","Type":"ContainerDied","Data":"8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db"} Nov 03 09:56:36.667674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.667627 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" Nov 03 09:56:36.667674 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.667675 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8" event={"ID":"4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb","Type":"ContainerDied","Data":"31a3e5982d41f4b68b584b3506a50fb907d4c317ccbead160a1cf76c0f96d7b5"} Nov 03 09:56:36.668017 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.667702 5015 scope.go:122] "RemoveContainer" containerID="8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db" Nov 03 09:56:36.669745 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.669717 5015 generic.go:356] "Generic (PLEG): container finished" podID="20ad04c6-7102-4548-b311-81690a614cc9" containerID="8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072" exitCode=0 Nov 03 09:56:36.669892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.669820 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" event={"ID":"20ad04c6-7102-4548-b311-81690a614cc9","Type":"ContainerDied","Data":"8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072"} Nov 03 09:56:36.669892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.669866 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" event={"ID":"20ad04c6-7102-4548-b311-81690a614cc9","Type":"ContainerDied","Data":"8ff04633091591c9e825363efb6bb36cbefaeec27d81458e33579386b38e70a0"} Nov 03 09:56:36.670007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.669943 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7" Nov 03 09:56:36.689204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.689175 5015 scope.go:122] "RemoveContainer" containerID="8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db" Nov 03 09:56:36.691498 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:36.691374 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db\": container with ID starting with 8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db not found: ID does not exist" containerID="8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db" Nov 03 09:56:36.691498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.691415 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db"} err="failed to get container status \"8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db\": rpc error: code = NotFound desc = could not find container \"8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db\": container with ID starting with 8c9a72a2fb4c2748ce7ef0dc9ab561caa3b3e162a5fedca1158b2c7eeb9558db not found: ID does not exist" Nov 03 09:56:36.691498 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.691441 5015 scope.go:122] "RemoveContainer" containerID="8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072" Nov 03 09:56:36.708332 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.708309 5015 scope.go:122] "RemoveContainer" containerID="8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072" Nov 03 09:56:36.708659 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:36.708635 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072\": container with ID starting with 8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072 not found: ID does not exist" containerID="8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072" Nov 03 09:56:36.708727 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.708669 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072"} err="failed to get container status \"8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072\": rpc error: code = NotFound desc = could not find container \"8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072\": container with ID starting with 8ce2a006ba79eb8c138ad6bf15927098b7512180a1e926be5fcb2dc4e43fb072 not found: ID does not exist" Nov 03 09:56:36.839038 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.838996 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8"] Nov 03 09:56:36.889312 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.889275 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9ed110-7f6b84f77f-ggqh8"] Nov 03 09:56:36.957095 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.957007 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7"] Nov 03 09:56:36.963910 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:36.963880 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9ed110-76648b99-zxkc7"] Nov 03 09:56:37.205518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:37.205465 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ad04c6-7102-4548-b311-81690a614cc9" path="/var/lib/kubelet/pods/20ad04c6-7102-4548-b311-81690a614cc9/volumes" Nov 03 09:56:37.205935 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:37.205920 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb" path="/var/lib/kubelet/pods/4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb/volumes" Nov 03 09:56:40.020181 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:40.020151 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6"] Nov 03 09:56:40.020702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:40.020359 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" containerID="cri-o://79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05" gracePeriod=30 Nov 03 09:56:40.830791 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:40.830751 5015 patch_prober.go:29] interesting pod/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:56:40.831010 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:40.830813 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:56:40.900748 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:40.900703 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r"] Nov 03 09:56:40.932428 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:40.932392 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654"] Nov 03 09:56:41.134421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:41.134325 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" podUID="eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:56:41.134421 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:41.134374 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" podUID="1cc5a185-5f49-47f1-a634-65346de5e8b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:56:45.297685 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.297649 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct"] Nov 03 09:56:45.298214 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.297902 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" containerID="cri-o://4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb" gracePeriod=30 Nov 03 09:56:45.303320 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.303259 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:56:45.310383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.307398 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb"] Nov 03 09:56:45.310383 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.307614 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" containerID="cri-o://01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203" gracePeriod=30 Nov 03 09:56:45.388243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.388208 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:56:45.388401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.388264 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:56:45.446650 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.446610 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-data\") pod \"63a8c80d-0637-4e7f-81b9-e17e2523d291\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " Nov 03 09:56:45.446881 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.446665 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-envoy\") pod \"63a8c80d-0637-4e7f-81b9-e17e2523d291\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " Nov 03 09:56:45.446881 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.446815 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-socket\") pod \"63a8c80d-0637-4e7f-81b9-e17e2523d291\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " Nov 03 09:56:45.446881 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.446855 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-podinfo\") pod \"63a8c80d-0637-4e7f-81b9-e17e2523d291\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " Nov 03 09:56:45.447079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.446900 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-credential-socket\") pod \"63a8c80d-0637-4e7f-81b9-e17e2523d291\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " Nov 03 09:56:45.447079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.446934 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-token\" (UniqueName: \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-token\") pod \"63a8c80d-0637-4e7f-81b9-e17e2523d291\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " Nov 03 09:56:45.447079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.446958 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-data" pod "63a8c80d-0637-4e7f-81b9-e17e2523d291" (UID: "63a8c80d-0637-4e7f-81b9-e17e2523d291"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:45.447079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.446965 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-kube-api-access-7hk5h\" (UniqueName: \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-kube-api-access-7hk5h\") pod \"63a8c80d-0637-4e7f-81b9-e17e2523d291\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " Nov 03 09:56:45.447079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.447001 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a8c80d-0637-4e7f-81b9-e17e2523d291-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a8c80d-0637-4e7f-81b9-e17e2523d291-istiod-ca-cert\") pod \"63a8c80d-0637-4e7f-81b9-e17e2523d291\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " Nov 03 09:56:45.447079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.447035 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-certs\") pod \"63a8c80d-0637-4e7f-81b9-e17e2523d291\" (UID: \"63a8c80d-0637-4e7f-81b9-e17e2523d291\") " Nov 03 09:56:45.447413 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.447355 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-socket" pod "63a8c80d-0637-4e7f-81b9-e17e2523d291" (UID: "63a8c80d-0637-4e7f-81b9-e17e2523d291"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:45.448527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.448455 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-certs" pod "63a8c80d-0637-4e7f-81b9-e17e2523d291" (UID: "63a8c80d-0637-4e7f-81b9-e17e2523d291"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:45.448741 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.448710 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-credential-socket" pod "63a8c80d-0637-4e7f-81b9-e17e2523d291" (UID: "63a8c80d-0637-4e7f-81b9-e17e2523d291"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:45.448997 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.448966 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a8c80d-0637-4e7f-81b9-e17e2523d291-istiod-ca-cert" pod "63a8c80d-0637-4e7f-81b9-e17e2523d291" (UID: "63a8c80d-0637-4e7f-81b9-e17e2523d291"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:45.449290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.449197 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:45.449290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.449220 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:45.449290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.449237 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a8c80d-0637-4e7f-81b9-e17e2523d291-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:45.449290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.449253 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:45.449290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.449270 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:45.449942 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.449897 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-podinfo" pod "63a8c80d-0637-4e7f-81b9-e17e2523d291" (UID: "63a8c80d-0637-4e7f-81b9-e17e2523d291"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:56:45.455528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.451526 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-token" pod "63a8c80d-0637-4e7f-81b9-e17e2523d291" (UID: "63a8c80d-0637-4e7f-81b9-e17e2523d291"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:45.455528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.451558 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-envoy" pod "63a8c80d-0637-4e7f-81b9-e17e2523d291" (UID: "63a8c80d-0637-4e7f-81b9-e17e2523d291"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:45.455528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.451767 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-kube-api-access-7hk5h" pod "63a8c80d-0637-4e7f-81b9-e17e2523d291" (UID: "63a8c80d-0637-4e7f-81b9-e17e2523d291"). InnerVolumeSpecName "kube-api-access-7hk5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:45.550523 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.550413 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:45.550523 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.550441 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:45.550523 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.550451 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hk5h\" (UniqueName: \"kubernetes.io/projected/63a8c80d-0637-4e7f-81b9-e17e2523d291-kube-api-access-7hk5h\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:45.550523 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.550462 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a8c80d-0637-4e7f-81b9-e17e2523d291-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:45.733143 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.733108 5015 generic.go:356] "Generic (PLEG): container finished" podID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerID="79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05" exitCode=0 Nov 03 09:56:45.733325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.733162 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" Nov 03 09:56:45.733325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.733195 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" event={"ID":"63a8c80d-0637-4e7f-81b9-e17e2523d291","Type":"ContainerDied","Data":"79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05"} Nov 03 09:56:45.733325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.733252 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6" event={"ID":"63a8c80d-0637-4e7f-81b9-e17e2523d291","Type":"ContainerDied","Data":"edb88845d49fcf11990bad2fc5e5ae035f9dfa0531ac7514164ca8b455a93aa9"} Nov 03 09:56:45.733325 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.733281 5015 scope.go:122] "RemoveContainer" containerID="79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05" Nov 03 09:56:45.744469 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.744452 5015 scope.go:122] "RemoveContainer" containerID="957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82" Nov 03 09:56:45.754001 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.753978 5015 scope.go:122] "RemoveContainer" containerID="79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05" Nov 03 09:56:45.754276 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:45.754255 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05\": container with ID starting with 79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05 not found: ID does not exist" containerID="79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05" Nov 03 09:56:45.754348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.754284 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05"} err="failed to get container status \"79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05\": rpc error: code = NotFound desc = could not find container \"79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05\": container with ID starting with 79736c345870c904d1b2d02d90082ae580a9beb81333453bc3dc503ee0773d05 not found: ID does not exist" Nov 03 09:56:45.754348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.754302 5015 scope.go:122] "RemoveContainer" containerID="957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82" Nov 03 09:56:45.754590 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:45.754569 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82\": container with ID starting with 957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82 not found: ID does not exist" containerID="957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82" Nov 03 09:56:45.754683 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.754596 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82"} err="failed to get container status \"957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82\": rpc error: code = NotFound desc = could not find container \"957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82\": container with ID starting with 957baa12efd0894c465da6a72c34d6f128fde14d6245bec7aac46d050f552d82 not found: ID does not exist" Nov 03 09:56:45.903433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.903345 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6"] Nov 03 09:56:45.912608 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:45.912581 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-627e5f-openshift-default-5b4984b4ff-7j2l6"] Nov 03 09:56:46.902542 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:46.902468 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:56:47.204719 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:47.204640 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" path="/var/lib/kubelet/pods/63a8c80d-0637-4e7f-81b9-e17e2523d291/volumes" Nov 03 09:56:48.248767 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.248735 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v"] Nov 03 09:56:48.358300 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.358219 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9"] Nov 03 09:56:48.733317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733285 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb"] Nov 03 09:56:48.733817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733802 5015 cpu_manager.go:414] "containerMap: removing container" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733819 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733830 5015 cpu_manager.go:414] "containerMap: removing container" podUID="850f7210-fa1a-409c-b999-c76ffb9de8ad" containerName="maas-api" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733835 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850f7210-fa1a-409c-b999-c76ffb9de8ad" containerName="maas-api" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733848 5015 cpu_manager.go:414] "containerMap: removing container" podUID="d353fe68-ba7d-433c-b7b6-9dca79d7de10" containerName="payload-pre-processing" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733853 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d353fe68-ba7d-433c-b7b6-9dca79d7de10" containerName="payload-pre-processing" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733863 5015 cpu_manager.go:414] "containerMap: removing container" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733868 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733875 5015 cpu_manager.go:414] "containerMap: removing container" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733880 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733886 5015 cpu_manager.go:414] "containerMap: removing container" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733890 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733908 5015 cpu_manager.go:414] "containerMap: removing container" podUID="20ad04c6-7102-4548-b311-81690a614cc9" containerName="payload-pre-processing" Nov 03 09:56:48.733911 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733912 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ad04c6-7102-4548-b311-81690a614cc9" containerName="payload-pre-processing" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733924 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a39c892d-ff3a-43d4-bcad-c77218158498" containerName="maas-api" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733928 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39c892d-ff3a-43d4-bcad-c77218158498" containerName="maas-api" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733935 5015 cpu_manager.go:414] "containerMap: removing container" podUID="ed11b2ca-3e57-40d1-8eeb-20a31f9533c2" containerName="maas-api" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733939 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed11b2ca-3e57-40d1-8eeb-20a31f9533c2" containerName="maas-api" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733946 5015 cpu_manager.go:414] "containerMap: removing container" podUID="4e5255ed-f323-4c2f-b431-14b4bcf956e4" containerName="payload-processing" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733951 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5255ed-f323-4c2f-b431-14b4bcf956e4" containerName="payload-processing" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733961 5015 cpu_manager.go:414] "containerMap: removing container" podUID="4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb" containerName="payload-processing" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733966 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb" containerName="payload-processing" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733974 5015 cpu_manager.go:414] "containerMap: removing container" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.733978 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734043 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734051 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5255ed-f323-4c2f-b431-14b4bcf956e4" containerName="payload-processing" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734057 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa279c1-6afa-4a7e-b57c-8f3fbd95cfdb" containerName="payload-processing" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734064 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bae820b-3e21-4df3-be7f-7fc15df83f8c" containerName="istio-proxy" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734071 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734076 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="850f7210-fa1a-409c-b999-c76ffb9de8ad" containerName="maas-api" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734081 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="186ae8ad-21e6-4fde-afaa-549164cc1b1d" containerName="istio-proxy" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734087 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="d353fe68-ba7d-433c-b7b6-9dca79d7de10" containerName="payload-pre-processing" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734093 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a8c80d-0637-4e7f-81b9-e17e2523d291" containerName="istio-proxy" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734100 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ad04c6-7102-4548-b311-81690a614cc9" containerName="payload-pre-processing" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734105 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39c892d-ff3a-43d4-bcad-c77218158498" containerName="maas-api" Nov 03 09:56:48.734323 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.734111 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed11b2ca-3e57-40d1-8eeb-20a31f9533c2" containerName="maas-api" Nov 03 09:56:48.738931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.738908 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:48.794958 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.794879 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:56:48.803532 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.800724 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb"] Nov 03 09:56:48.803532 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.801510 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 09:56:48.888880 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.888845 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b61c7df-pbzkb\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:48.889088 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.888886 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b61c7df-pbzkb\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:48.889088 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.888940 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg4dm\" (UniqueName: \"kubernetes.io/projected/55a4c0d0-ad33-41df-ac33-99f23537a233-kube-api-access-wg4dm\") pod \"maas-api-revoke-sub-0b61c7df-pbzkb\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:48.990706 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.990613 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b61c7df-pbzkb\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:48.990706 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.990652 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b61c7df-pbzkb\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:48.990706 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.990679 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg4dm\" (UniqueName: \"kubernetes.io/projected/55a4c0d0-ad33-41df-ac33-99f23537a233-kube-api-access-wg4dm\") pod \"maas-api-revoke-sub-0b61c7df-pbzkb\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:48.991406 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.991383 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b61c7df-pbzkb\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:48.993541 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:48.993518 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b61c7df-pbzkb\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:49.050222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.050188 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg4dm\" (UniqueName: \"kubernetes.io/projected/55a4c0d0-ad33-41df-ac33-99f23537a233-kube-api-access-wg4dm\") pod \"maas-api-revoke-sub-0b61c7df-pbzkb\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:49.053023 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.052991 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:49.437205 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.437173 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb"] Nov 03 09:56:49.544305 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.544270 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:56:49.658974 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.658945 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:56:49.701557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.701514 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-kube-api-access-lzk4k\" (UniqueName: \"kubernetes.io/projected/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-kube-api-access-lzk4k\") pod \"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1\" (UID: \"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1\") " Nov 03 09:56:49.701756 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.701678 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-plugins-config-volume\") pod \"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1\" (UID: \"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1\") " Nov 03 09:56:49.702029 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.702005 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-plugins-config-volume" pod "eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1" (UID: "eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:49.704087 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.704064 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-kube-api-access-lzk4k" pod "eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1" (UID: "eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1"). InnerVolumeSpecName "kube-api-access-lzk4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:49.766431 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.766400 5015 generic.go:356] "Generic (PLEG): container finished" podID="eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1" containerID="3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82" exitCode=0 Nov 03 09:56:49.766584 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.766465 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" Nov 03 09:56:49.766584 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.766467 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" event={"ID":"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1","Type":"ContainerDied","Data":"3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82"} Nov 03 09:56:49.766699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.766635 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w" event={"ID":"eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1","Type":"ContainerDied","Data":"6acd33cfcae82503f9304d4cfb69f8d215d35da5b12f25e4b5ca4d6a77f1c132"} Nov 03 09:56:49.766699 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.766671 5015 scope.go:122] "RemoveContainer" containerID="3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82" Nov 03 09:56:49.768843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.768817 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" event={"ID":"55a4c0d0-ad33-41df-ac33-99f23537a233","Type":"ContainerStarted","Data":"ff7a00fab6e6a045b3ef322d08c9f4f00b54c09c80c4ea7314347aacd6b8f7b7"} Nov 03 09:56:49.768978 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.768867 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" event={"ID":"55a4c0d0-ad33-41df-ac33-99f23537a233","Type":"ContainerStarted","Data":"fe5b8b41027a686303ea91ebde37b4e5d30c2da751e9b251b1a938a467811ede"} Nov 03 09:56:49.770401 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.770377 5015 generic.go:356] "Generic (PLEG): container finished" podID="1cc5a185-5f49-47f1-a634-65346de5e8b8" containerID="4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7" exitCode=0 Nov 03 09:56:49.770566 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.770424 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" Nov 03 09:56:49.770566 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.770435 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" event={"ID":"1cc5a185-5f49-47f1-a634-65346de5e8b8","Type":"ContainerDied","Data":"4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7"} Nov 03 09:56:49.770566 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.770506 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb" event={"ID":"1cc5a185-5f49-47f1-a634-65346de5e8b8","Type":"ContainerDied","Data":"e254a6f2b8389ab5ab2f92c5174ff7d4947b29b9bf050eb462ebdf3d355bd4e2"} Nov 03 09:56:49.807167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.802376 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc5a185-5f49-47f1-a634-65346de5e8b8-kube-api-access-lmcmx\" (UniqueName: \"kubernetes.io/projected/1cc5a185-5f49-47f1-a634-65346de5e8b8-kube-api-access-lmcmx\") pod \"1cc5a185-5f49-47f1-a634-65346de5e8b8\" (UID: \"1cc5a185-5f49-47f1-a634-65346de5e8b8\") " Nov 03 09:56:49.807167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.802730 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cc5a185-5f49-47f1-a634-65346de5e8b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc5a185-5f49-47f1-a634-65346de5e8b8-plugins-config-volume\") pod \"1cc5a185-5f49-47f1-a634-65346de5e8b8\" (UID: \"1cc5a185-5f49-47f1-a634-65346de5e8b8\") " Nov 03 09:56:49.807167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.803182 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:49.807167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.803202 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzk4k\" (UniqueName: \"kubernetes.io/projected/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1-kube-api-access-lzk4k\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:49.807167 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.803703 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc5a185-5f49-47f1-a634-65346de5e8b8-plugins-config-volume" pod "1cc5a185-5f49-47f1-a634-65346de5e8b8" (UID: "1cc5a185-5f49-47f1-a634-65346de5e8b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:49.812079 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.812048 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc5a185-5f49-47f1-a634-65346de5e8b8-kube-api-access-lmcmx" pod "1cc5a185-5f49-47f1-a634-65346de5e8b8" (UID: "1cc5a185-5f49-47f1-a634-65346de5e8b8"). InnerVolumeSpecName "kube-api-access-lmcmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:49.820746 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.820722 5015 scope.go:122] "RemoveContainer" containerID="3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82" Nov 03 09:56:49.821081 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:49.821054 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82\": container with ID starting with 3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82 not found: ID does not exist" containerID="3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82" Nov 03 09:56:49.821196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.821087 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82"} err="failed to get container status \"3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82\": rpc error: code = NotFound desc = could not find container \"3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82\": container with ID starting with 3f0eaf39d563d9dd93d57103e8f242f339e575623dcb53414b67f848d0c9dc82 not found: ID does not exist" Nov 03 09:56:49.821196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.821105 5015 scope.go:122] "RemoveContainer" containerID="4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7" Nov 03 09:56:49.831027 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.831000 5015 scope.go:122] "RemoveContainer" containerID="4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7" Nov 03 09:56:49.831337 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:49.831317 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7\": container with ID starting with 4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7 not found: ID does not exist" containerID="4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7" Nov 03 09:56:49.831494 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.831347 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7"} err="failed to get container status \"4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7\": rpc error: code = NotFound desc = could not find container \"4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7\": container with ID starting with 4e8aa54dad9ae8fed40f74b02f96a1af164f34f1bc54a9c2f3125cb3d8e914c7 not found: ID does not exist" Nov 03 09:56:49.904598 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.904563 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc5a185-5f49-47f1-a634-65346de5e8b8-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:49.904598 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.904598 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmcmx\" (UniqueName: \"kubernetes.io/projected/1cc5a185-5f49-47f1-a634-65346de5e8b8-kube-api-access-lmcmx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:49.949230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:49.949172 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" podStartSLOduration=1.9491580659999999 podStartE2EDuration="1.949158066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:56:49.947470314 +0000 UTC m=+4642.872412559" watchObservedRunningTime="2026-11-03 09:56:49.949158066 +0000 UTC m=+4642.874100282" Nov 03 09:56:50.118931 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.118893 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w"] Nov 03 09:56:50.125558 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.125523 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-627e5f-f44945f57-t2n9w"] Nov 03 09:56:50.250622 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.250586 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb"] Nov 03 09:56:50.265044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.265011 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-627e5f-5998fdfff9-5jsqb"] Nov 03 09:56:50.590689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.590662 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:56:50.601260 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.601237 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:56:50.714770 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.714679 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-podinfo\") pod \"94ab4552-a914-4ccc-a3e1-211981c0feae\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " Nov 03 09:56:50.714770 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.714760 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-kube-api-access-4775d\" (UniqueName: \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-kube-api-access-4775d\") pod \"94ab4552-a914-4ccc-a3e1-211981c0feae\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " Nov 03 09:56:50.714969 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.714792 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-credential-socket\") pod \"94ab4552-a914-4ccc-a3e1-211981c0feae\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " Nov 03 09:56:50.714969 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.714855 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ab4552-a914-4ccc-a3e1-211981c0feae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab4552-a914-4ccc-a3e1-211981c0feae-istiod-ca-cert\") pod \"94ab4552-a914-4ccc-a3e1-211981c0feae\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " Nov 03 09:56:50.714969 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.714901 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-token\" (UniqueName: \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-token\") pod \"94ab4552-a914-4ccc-a3e1-211981c0feae\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " Nov 03 09:56:50.714969 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.714957 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-certs\") pod \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " Nov 03 09:56:50.715154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.714986 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-socket\") pod \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " Nov 03 09:56:50.715154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.715016 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-podinfo\") pod \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " Nov 03 09:56:50.715154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.715046 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-kube-api-access-8w8ps\" (UniqueName: \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-kube-api-access-8w8ps\") pod \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " Nov 03 09:56:50.715154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.715088 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-envoy\") pod \"94ab4552-a914-4ccc-a3e1-211981c0feae\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " Nov 03 09:56:50.715154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.715115 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-certs\") pod \"94ab4552-a914-4ccc-a3e1-211981c0feae\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " Nov 03 09:56:50.715154 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.715143 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-credential-socket\") pod \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " Nov 03 09:56:50.715381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.715174 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-data\") pod \"94ab4552-a914-4ccc-a3e1-211981c0feae\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " Nov 03 09:56:50.715381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.715260 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-data\") pod \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " Nov 03 09:56:50.715381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.715294 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-envoy\") pod \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " Nov 03 09:56:50.715381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.715333 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-socket\") pod \"94ab4552-a914-4ccc-a3e1-211981c0feae\" (UID: \"94ab4552-a914-4ccc-a3e1-211981c0feae\") " Nov 03 09:56:50.715381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.715364 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-token\" (UniqueName: \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-token\") pod \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " Nov 03 09:56:50.715658 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.715408 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istiod-ca-cert\") pod \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\" (UID: \"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f\") " Nov 03 09:56:50.716228 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.716198 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-credential-socket" pod "94ab4552-a914-4ccc-a3e1-211981c0feae" (UID: "94ab4552-a914-4ccc-a3e1-211981c0feae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:50.716432 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.716181 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-certs" pod "94ab4552-a914-4ccc-a3e1-211981c0feae" (UID: "94ab4552-a914-4ccc-a3e1-211981c0feae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:50.716567 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.716222 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-data" pod "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" (UID: "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:50.716665 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.716273 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istiod-ca-cert" pod "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" (UID: "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:50.716755 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.716421 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-certs" pod "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" (UID: "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:50.716847 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.716826 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-socket" pod "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" (UID: "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:50.716915 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.716460 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-credential-socket" pod "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" (UID: "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:50.716915 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.716661 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-data" pod "94ab4552-a914-4ccc-a3e1-211981c0feae" (UID: "94ab4552-a914-4ccc-a3e1-211981c0feae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:50.717025 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.716996 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-socket" pod "94ab4552-a914-4ccc-a3e1-211981c0feae" (UID: "94ab4552-a914-4ccc-a3e1-211981c0feae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:50.717430 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.717403 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ab4552-a914-4ccc-a3e1-211981c0feae-istiod-ca-cert" pod "94ab4552-a914-4ccc-a3e1-211981c0feae" (UID: "94ab4552-a914-4ccc-a3e1-211981c0feae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:50.717613 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.717574 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-podinfo" pod "94ab4552-a914-4ccc-a3e1-211981c0feae" (UID: "94ab4552-a914-4ccc-a3e1-211981c0feae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:56:50.718026 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.717883 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-token" pod "94ab4552-a914-4ccc-a3e1-211981c0feae" (UID: "94ab4552-a914-4ccc-a3e1-211981c0feae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:50.719707 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.719671 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-kube-api-access-4775d" pod "94ab4552-a914-4ccc-a3e1-211981c0feae" (UID: "94ab4552-a914-4ccc-a3e1-211981c0feae"). InnerVolumeSpecName "kube-api-access-4775d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:50.720021 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.719998 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-envoy" pod "94ab4552-a914-4ccc-a3e1-211981c0feae" (UID: "94ab4552-a914-4ccc-a3e1-211981c0feae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:50.720326 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.720305 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-kube-api-access-8w8ps" pod "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" (UID: "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f"). InnerVolumeSpecName "kube-api-access-8w8ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:50.720848 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.720826 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-token" pod "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" (UID: "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:50.721557 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.721532 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-podinfo" pod "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" (UID: "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:56:50.721661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.721641 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-envoy" pod "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" (UID: "86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:50.781059 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.781023 5015 generic.go:356] "Generic (PLEG): container finished" podID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerID="4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb" exitCode=0 Nov 03 09:56:50.781248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.781086 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" Nov 03 09:56:50.781248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.781109 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" event={"ID":"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f","Type":"ContainerDied","Data":"4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb"} Nov 03 09:56:50.781248 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.781215 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct" event={"ID":"86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f","Type":"ContainerDied","Data":"d2e5a5b2efe5a5e5d000eaafb28bd87a56da3c8bdede84fe8a969829134b1ad4"} Nov 03 09:56:50.781433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.781256 5015 scope.go:122] "RemoveContainer" containerID="4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb" Nov 03 09:56:50.784626 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.784598 5015 generic.go:356] "Generic (PLEG): container finished" podID="55a4c0d0-ad33-41df-ac33-99f23537a233" containerID="ff7a00fab6e6a045b3ef322d08c9f4f00b54c09c80c4ea7314347aacd6b8f7b7" exitCode=0 Nov 03 09:56:50.784784 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.784691 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" event={"ID":"55a4c0d0-ad33-41df-ac33-99f23537a233","Type":"ContainerDied","Data":"ff7a00fab6e6a045b3ef322d08c9f4f00b54c09c80c4ea7314347aacd6b8f7b7"} Nov 03 09:56:50.786403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.786378 5015 generic.go:356] "Generic (PLEG): container finished" podID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerID="01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203" exitCode=0 Nov 03 09:56:50.786561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.786449 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" Nov 03 09:56:50.786561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.786496 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" event={"ID":"94ab4552-a914-4ccc-a3e1-211981c0feae","Type":"ContainerDied","Data":"01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203"} Nov 03 09:56:50.786561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.786547 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb" event={"ID":"94ab4552-a914-4ccc-a3e1-211981c0feae","Type":"ContainerDied","Data":"d9d53e84e4944197cf59add88e5715a2343e6c17a75bfdb9b0d0f93b24686e77"} Nov 03 09:56:50.793155 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.793133 5015 scope.go:122] "RemoveContainer" containerID="77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb" Nov 03 09:56:50.803006 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.802989 5015 scope.go:122] "RemoveContainer" containerID="4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb" Nov 03 09:56:50.803302 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:50.803277 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb\": container with ID starting with 4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb not found: ID does not exist" containerID="4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb" Nov 03 09:56:50.803398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.803310 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb"} err="failed to get container status \"4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb\": rpc error: code = NotFound desc = could not find container \"4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb\": container with ID starting with 4318ba660b9a2a092c720f2cde395aaf4ec3f5a8ac23f8c283bc6050af4fb5eb not found: ID does not exist" Nov 03 09:56:50.803398 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.803335 5015 scope.go:122] "RemoveContainer" containerID="77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb" Nov 03 09:56:50.803649 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:50.803627 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb\": container with ID starting with 77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb not found: ID does not exist" containerID="77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb" Nov 03 09:56:50.803687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.803656 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb"} err="failed to get container status \"77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb\": rpc error: code = NotFound desc = could not find container \"77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb\": container with ID starting with 77cafe69d1cebc1de42482e9a35c22c59679ed5ae6372208dacfbad3ab932abb not found: ID does not exist" Nov 03 09:56:50.803687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.803671 5015 scope.go:122] "RemoveContainer" containerID="01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203" Nov 03 09:56:50.813929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.813914 5015 scope.go:122] "RemoveContainer" containerID="01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203" Nov 03 09:56:50.814187 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:50.814170 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203\": container with ID starting with 01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203 not found: ID does not exist" containerID="01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203" Nov 03 09:56:50.814242 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.814193 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203"} err="failed to get container status \"01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203\": rpc error: code = NotFound desc = could not find container \"01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203\": container with ID starting with 01fe65b76edf7fe833ede6d0147844ac8fa816ae0339031fc1d8fe3b6d4d5203 not found: ID does not exist" Nov 03 09:56:50.816884 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.816853 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.816884 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.816878 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.816895 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.816907 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.816920 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.816933 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.816948 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4775d\" (UniqueName: \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-kube-api-access-4775d\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.816963 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.816976 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab4552-a914-4ccc-a3e1-211981c0feae-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.816990 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.817003 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.817017 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.817031 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.817046 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8ps\" (UniqueName: \"kubernetes.io/projected/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-kube-api-access-8w8ps\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817448 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.817060 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817448 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.817073 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817448 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.817087 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:50.817448 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:50.817102 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab4552-a914-4ccc-a3e1-211981c0feae-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:51.068943 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:51.068906 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb"] Nov 03 09:56:51.073198 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:51.073169 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pgwkb"] Nov 03 09:56:51.182942 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:51.182906 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct"] Nov 03 09:56:51.189788 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:51.189755 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9ed110-openshift-default-75dc7b494f-rs4ct"] Nov 03 09:56:51.204391 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:51.204364 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc5a185-5f49-47f1-a634-65346de5e8b8" path="/var/lib/kubelet/pods/1cc5a185-5f49-47f1-a634-65346de5e8b8/volumes" Nov 03 09:56:51.204906 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:51.204890 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" path="/var/lib/kubelet/pods/86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f/volumes" Nov 03 09:56:51.205319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:51.205306 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" path="/var/lib/kubelet/pods/94ab4552-a914-4ccc-a3e1-211981c0feae/volumes" Nov 03 09:56:51.205833 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:51.205813 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1" path="/var/lib/kubelet/pods/eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1/volumes" Nov 03 09:56:51.958518 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:51.958471 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:52.131071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.130983 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-openshift-ca\") pod \"55a4c0d0-ad33-41df-ac33-99f23537a233\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " Nov 03 09:56:52.131071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.131030 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a4c0d0-ad33-41df-ac33-99f23537a233-kube-api-access-wg4dm\" (UniqueName: \"kubernetes.io/projected/55a4c0d0-ad33-41df-ac33-99f23537a233-kube-api-access-wg4dm\") pod \"55a4c0d0-ad33-41df-ac33-99f23537a233\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " Nov 03 09:56:52.131071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.131070 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-xks-ca\") pod \"55a4c0d0-ad33-41df-ac33-99f23537a233\" (UID: \"55a4c0d0-ad33-41df-ac33-99f23537a233\") " Nov 03 09:56:52.131392 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.131339 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-openshift-ca" pod "55a4c0d0-ad33-41df-ac33-99f23537a233" (UID: "55a4c0d0-ad33-41df-ac33-99f23537a233"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:52.133572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.133536 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-xks-ca" pod "55a4c0d0-ad33-41df-ac33-99f23537a233" (UID: "55a4c0d0-ad33-41df-ac33-99f23537a233"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:56:52.133572 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.133556 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a4c0d0-ad33-41df-ac33-99f23537a233-kube-api-access-wg4dm" pod "55a4c0d0-ad33-41df-ac33-99f23537a233" (UID: "55a4c0d0-ad33-41df-ac33-99f23537a233"). InnerVolumeSpecName "kube-api-access-wg4dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:52.232032 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.231995 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:52.232032 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.232025 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg4dm\" (UniqueName: \"kubernetes.io/projected/55a4c0d0-ad33-41df-ac33-99f23537a233-kube-api-access-wg4dm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:52.232032 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.232035 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a4c0d0-ad33-41df-ac33-99f23537a233-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:52.803751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.803706 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" event={"ID":"55a4c0d0-ad33-41df-ac33-99f23537a233","Type":"ContainerDied","Data":"fe5b8b41027a686303ea91ebde37b4e5d30c2da751e9b251b1a938a467811ede"} Nov 03 09:56:52.803751 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.803743 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb" Nov 03 09:56:52.803980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:52.803805 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe5b8b41027a686303ea91ebde37b4e5d30c2da751e9b251b1a938a467811ede" Nov 03 09:56:53.249221 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.249167 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" podUID="0bbca83d-8f02-412a-b508-2633ec103a94" containerName="payload-processing" containerID="cri-o://1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa" gracePeriod=25 Nov 03 09:56:53.509107 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.509051 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 09:56:53.555456 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.555422 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bbca83d-8f02-412a-b508-2633ec103a94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bbca83d-8f02-412a-b508-2633ec103a94-plugins-config-volume\") pod \"0bbca83d-8f02-412a-b508-2633ec103a94\" (UID: \"0bbca83d-8f02-412a-b508-2633ec103a94\") " Nov 03 09:56:53.555628 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.555466 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bbca83d-8f02-412a-b508-2633ec103a94-kube-api-access-l7rlz\" (UniqueName: \"kubernetes.io/projected/0bbca83d-8f02-412a-b508-2633ec103a94-kube-api-access-l7rlz\") pod \"0bbca83d-8f02-412a-b508-2633ec103a94\" (UID: \"0bbca83d-8f02-412a-b508-2633ec103a94\") " Nov 03 09:56:53.555912 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.555882 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbca83d-8f02-412a-b508-2633ec103a94-plugins-config-volume" pod "0bbca83d-8f02-412a-b508-2633ec103a94" (UID: "0bbca83d-8f02-412a-b508-2633ec103a94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:53.557906 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.557881 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbca83d-8f02-412a-b508-2633ec103a94-kube-api-access-l7rlz" pod "0bbca83d-8f02-412a-b508-2633ec103a94" (UID: "0bbca83d-8f02-412a-b508-2633ec103a94"). InnerVolumeSpecName "kube-api-access-l7rlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:53.658789 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.658753 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bbca83d-8f02-412a-b508-2633ec103a94-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:53.658789 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.658791 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7rlz\" (UniqueName: \"kubernetes.io/projected/0bbca83d-8f02-412a-b508-2633ec103a94-kube-api-access-l7rlz\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:53.756024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.755971 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" podUID="c4df8dd8-b208-42c5-b6ef-fdc924243ba6" containerName="payload-pre-processing" containerID="cri-o://b490bd341aa36a451c32c48403eeb41c132f6600df744c7273eb31a9c51ca0c8" gracePeriod=25 Nov 03 09:56:53.811066 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.810975 5015 generic.go:356] "Generic (PLEG): container finished" podID="0bbca83d-8f02-412a-b508-2633ec103a94" containerID="1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa" exitCode=0 Nov 03 09:56:53.811066 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.811023 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" Nov 03 09:56:53.811316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.811063 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" event={"ID":"0bbca83d-8f02-412a-b508-2633ec103a94","Type":"ContainerDied","Data":"1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa"} Nov 03 09:56:53.811316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.811125 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v" event={"ID":"0bbca83d-8f02-412a-b508-2633ec103a94","Type":"ContainerDied","Data":"e79ae7c7fa87e14405db27fd132303694b220423f4e53b0eb83daaffdb7559a4"} Nov 03 09:56:53.811316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.811153 5015 scope.go:122] "RemoveContainer" containerID="1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa" Nov 03 09:56:53.821204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.821187 5015 scope.go:122] "RemoveContainer" containerID="1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa" Nov 03 09:56:53.821500 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:53.821461 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa\": container with ID starting with 1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa not found: ID does not exist" containerID="1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa" Nov 03 09:56:53.821581 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.821508 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa"} err="failed to get container status \"1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa\": rpc error: code = NotFound desc = could not find container \"1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa\": container with ID starting with 1ca9c61cbf1336653aae3fd21be6716990d4aee605e87b08ff0b58f55a42bcfa not found: ID does not exist" Nov 03 09:56:53.980563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.980497 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v"] Nov 03 09:56:53.983158 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:53.983136 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-77207e-547f9c9675-s849v"] Nov 03 09:56:55.203563 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:55.203527 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bbca83d-8f02-412a-b508-2633ec103a94" path="/var/lib/kubelet/pods/0bbca83d-8f02-412a-b508-2633ec103a94/volumes" Nov 03 09:56:55.928989 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:55.928951 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" containerID="cri-o://02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57" gracePeriod=45 Nov 03 09:56:55.953389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:55.953359 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" containerID="cri-o://b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5" gracePeriod=45 Nov 03 09:56:56.211140 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.211112 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:56:56.234439 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.234416 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 09:56:56.283609 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283580 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-kube-api-access-f7s4q\" (UniqueName: \"kubernetes.io/projected/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-kube-api-access-f7s4q\") pod \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " Nov 03 09:56:56.283777 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283639 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tls-certs\") pod \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " Nov 03 09:56:56.283777 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283663 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tmp-dir\") pod \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " Nov 03 09:56:56.283777 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283685 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-home\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-home\") pod \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " Nov 03 09:56:56.283777 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283715 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-dshm\") pod \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " Nov 03 09:56:56.283777 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283755 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-model-cache\") pod \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " Nov 03 09:56:56.284054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283790 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-home\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-home\") pod \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " Nov 03 09:56:56.284054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283819 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-dshm\") pod \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " Nov 03 09:56:56.284054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283879 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tls-certs\" (UniqueName: \"kubernetes.io/secret/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tls-certs\") pod \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " Nov 03 09:56:56.284054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283922 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tmp-dir\") pod \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " Nov 03 09:56:56.284054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283968 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-kube-api-access-2kkz4\" (UniqueName: \"kubernetes.io/projected/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-kube-api-access-2kkz4\") pod \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\" (UID: \"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc\") " Nov 03 09:56:56.284054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.283993 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-model-cache\") pod \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\" (UID: \"56aad1c2-84bc-40fe-bba6-a0eb6374b47a\") " Nov 03 09:56:56.284054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.284002 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tmp-dir" pod "56aad1c2-84bc-40fe-bba6-a0eb6374b47a" (UID: "56aad1c2-84bc-40fe-bba6-a0eb6374b47a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.284054 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.284007 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-home" pod "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" (UID: "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.284502 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.284226 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tmp-dir" pod "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" (UID: "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.284986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.284652 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.284986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.284678 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.284986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.284691 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.284986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.284746 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-model-cache" pod "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" (UID: "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.284986 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.284969 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-home" pod "56aad1c2-84bc-40fe-bba6-a0eb6374b47a" (UID: "56aad1c2-84bc-40fe-bba6-a0eb6374b47a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.285345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.285022 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-model-cache" pod "56aad1c2-84bc-40fe-bba6-a0eb6374b47a" (UID: "56aad1c2-84bc-40fe-bba6-a0eb6374b47a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.286801 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.286770 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tls-certs" pod "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" (UID: "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:56:56.286902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.286815 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-dshm" pod "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" (UID: "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.286902 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.286860 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-dshm" pod "56aad1c2-84bc-40fe-bba6-a0eb6374b47a" (UID: "56aad1c2-84bc-40fe-bba6-a0eb6374b47a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.287033 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.287017 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-kube-api-access-f7s4q" pod "56aad1c2-84bc-40fe-bba6-a0eb6374b47a" (UID: "56aad1c2-84bc-40fe-bba6-a0eb6374b47a"). InnerVolumeSpecName "kube-api-access-f7s4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:56.287093 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.287045 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-kube-api-access-2kkz4" pod "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" (UID: "4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc"). InnerVolumeSpecName "kube-api-access-2kkz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:56.287267 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.287251 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tls-certs" pod "56aad1c2-84bc-40fe-bba6-a0eb6374b47a" (UID: "56aad1c2-84bc-40fe-bba6-a0eb6374b47a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:56:56.385890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.385843 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7s4q\" (UniqueName: \"kubernetes.io/projected/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-kube-api-access-f7s4q\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.385890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.385881 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.385890 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.385894 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.386120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.385905 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.386120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.385915 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.386120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.385923 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.386120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.385931 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.386120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.385940 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kkz4\" (UniqueName: \"kubernetes.io/projected/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc-kube-api-access-2kkz4\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.386120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.385948 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56aad1c2-84bc-40fe-bba6-a0eb6374b47a-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.415685 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.415647 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" podUID="c4df8dd8-b208-42c5-b6ef-fdc924243ba6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:56:56.845449 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.845414 5015 generic.go:356] "Generic (PLEG): container finished" podID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerID="b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5" exitCode=0 Nov 03 09:56:56.845679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.845459 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" Nov 03 09:56:56.845679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.845543 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" event={"ID":"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc","Type":"ContainerDied","Data":"b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5"} Nov 03 09:56:56.845679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.845590 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654" event={"ID":"4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc","Type":"ContainerDied","Data":"7af25e8c1b2f14e92be77f92ceb0972083f61e3ac1abefc4cd2b48d2fe4b2830"} Nov 03 09:56:56.845679 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.845607 5015 scope.go:122] "RemoveContainer" containerID="b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5" Nov 03 09:56:56.847279 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.847250 5015 generic.go:356] "Generic (PLEG): container finished" podID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerID="02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57" exitCode=0 Nov 03 09:56:56.847435 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.847289 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" event={"ID":"56aad1c2-84bc-40fe-bba6-a0eb6374b47a","Type":"ContainerDied","Data":"02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57"} Nov 03 09:56:56.847435 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.847334 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" event={"ID":"56aad1c2-84bc-40fe-bba6-a0eb6374b47a","Type":"ContainerDied","Data":"37f715123571e90555d8e1747f95c1db279d820a9998b4272546dab74718b842"} Nov 03 09:56:56.847435 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.847301 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r" Nov 03 09:56:56.855843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.855722 5015 scope.go:122] "RemoveContainer" containerID="b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5" Nov 03 09:56:56.856548 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:56.856367 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5\": container with ID starting with b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5 not found: ID does not exist" containerID="b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5" Nov 03 09:56:56.856548 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.856413 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5"} err="failed to get container status \"b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5\": rpc error: code = NotFound desc = could not find container \"b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5\": container with ID starting with b08107ebbe45ee5d5b61415f15f1e05e5b709ccc6475c2f7d73f58c0f5bc40d5 not found: ID does not exist" Nov 03 09:56:56.856548 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.856436 5015 scope.go:122] "RemoveContainer" containerID="02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57" Nov 03 09:56:56.866522 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.866505 5015 scope.go:122] "RemoveContainer" containerID="02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57" Nov 03 09:56:56.866782 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:56:56.866765 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57\": container with ID starting with 02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57 not found: ID does not exist" containerID="02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57" Nov 03 09:56:56.866843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:56.866792 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57"} err="failed to get container status \"02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57\": rpc error: code = NotFound desc = could not find container \"02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57\": container with ID starting with 02ad70efa1d29ee9a8737b33969cb944586697cdac5fbc6f0add26208598dc57 not found: ID does not exist" Nov 03 09:56:57.013610 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:57.013582 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r"] Nov 03 09:56:57.019254 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:57.019228 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-2-w2-77207e-kserve-6587649464-5s77r"] Nov 03 09:56:57.131175 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:57.131088 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654"] Nov 03 09:56:57.136404 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:57.136377 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/premium-w2-77207e-kserve-77d794f5d4-sl654"] Nov 03 09:56:57.203807 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:57.203783 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" path="/var/lib/kubelet/pods/4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc/volumes" Nov 03 09:56:57.204237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:57.204220 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" path="/var/lib/kubelet/pods/56aad1c2-84bc-40fe-bba6-a0eb6374b47a/volumes" Nov 03 09:56:58.167855 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:58.167821 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5"] Nov 03 09:56:58.292541 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:56:58.292506 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd"] Nov 03 09:57:00.620612 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:00.620576 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l"] Nov 03 09:57:01.752059 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:01.752020 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2"] Nov 03 09:57:01.752512 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:01.752243 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" podUID="190ecc81-1e36-4a1d-9b19-d1f9f2d06b15" containerName="maas-api" containerID="cri-o://a8eec25e0cd1079846ba08119a4d46ae8fe6c91f25069751573fa41db93b6a9b" gracePeriod=30 Nov 03 09:57:01.885680 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:01.885643 5015 generic.go:356] "Generic (PLEG): container finished" podID="190ecc81-1e36-4a1d-9b19-d1f9f2d06b15" containerID="a8eec25e0cd1079846ba08119a4d46ae8fe6c91f25069751573fa41db93b6a9b" exitCode=0 Nov 03 09:57:01.885881 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:01.885866 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" event={"ID":"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15","Type":"ContainerDied","Data":"a8eec25e0cd1079846ba08119a4d46ae8fe6c91f25069751573fa41db93b6a9b"} Nov 03 09:57:02.021094 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.021023 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:57:02.145501 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.145440 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-maas-api-tls\") pod \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " Nov 03 09:57:02.145720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.145562 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-kube-api-access-2vdlx\" (UniqueName: \"kubernetes.io/projected/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-kube-api-access-2vdlx\") pod \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " Nov 03 09:57:02.145720 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.145644 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-metrics-certs\") pod \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " Nov 03 09:57:02.145843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.145731 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-openshift-service-ca\") pod \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\" (UID: \"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15\") " Nov 03 09:57:02.146148 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.146100 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-openshift-service-ca" pod "190ecc81-1e36-4a1d-9b19-d1f9f2d06b15" (UID: "190ecc81-1e36-4a1d-9b19-d1f9f2d06b15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:02.148661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.148627 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-metrics-certs" pod "190ecc81-1e36-4a1d-9b19-d1f9f2d06b15" (UID: "190ecc81-1e36-4a1d-9b19-d1f9f2d06b15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:02.148661 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.148645 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-maas-api-tls" pod "190ecc81-1e36-4a1d-9b19-d1f9f2d06b15" (UID: "190ecc81-1e36-4a1d-9b19-d1f9f2d06b15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:02.148811 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.148656 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-kube-api-access-2vdlx" pod "190ecc81-1e36-4a1d-9b19-d1f9f2d06b15" (UID: "190ecc81-1e36-4a1d-9b19-d1f9f2d06b15"). InnerVolumeSpecName "kube-api-access-2vdlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:02.247689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.247646 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:02.247689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.247681 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vdlx\" (UniqueName: \"kubernetes.io/projected/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-kube-api-access-2vdlx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:02.247689 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.247695 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:02.247940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.247708 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:02.893049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.893013 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" Nov 03 09:57:02.893049 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.893020 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2" event={"ID":"190ecc81-1e36-4a1d-9b19-d1f9f2d06b15","Type":"ContainerDied","Data":"10c13d7a026351fe8c235aa43868ef3c41663a52c545e23db96a82ae40dac03b"} Nov 03 09:57:02.893667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:02.893079 5015 scope.go:122] "RemoveContainer" containerID="a8eec25e0cd1079846ba08119a4d46ae8fe6c91f25069751573fa41db93b6a9b" Nov 03 09:57:03.064863 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:03.064830 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2"] Nov 03 09:57:03.115843 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:03.115811 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b196c8-768478866d-xfmh2"] Nov 03 09:57:03.168627 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:03.168541 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" podUID="494d276c-7e71-4d57-a8b7-8cfc32b6b4c7" containerName="payload-processing" containerID="cri-o://79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca" gracePeriod=25 Nov 03 09:57:03.203852 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:03.203821 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190ecc81-1e36-4a1d-9b19-d1f9f2d06b15" path="/var/lib/kubelet/pods/190ecc81-1e36-4a1d-9b19-d1f9f2d06b15/volumes" Nov 03 09:57:03.292818 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:03.292772 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" podUID="bac4a5e6-5a35-47d9-9292-d1f1eb1c356f" containerName="payload-pre-processing" containerID="cri-o://1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3" gracePeriod=25 Nov 03 09:57:05.621156 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:05.621106 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" podUID="1c5b9e02-d908-4819-8d82-f2ce501928a7" containerName="payload-processing" containerID="cri-o://1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139" gracePeriod=25 Nov 03 09:57:06.416437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:06.416401 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" podUID="c4df8dd8-b208-42c5-b6ef-fdc924243ba6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:57:07.494041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.493998 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" podUID="494d276c-7e71-4d57-a8b7-8cfc32b6b4c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:57:07.494432 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.494058 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" podUID="bac4a5e6-5a35-47d9-9292-d1f1eb1c356f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:57:07.854757 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.854676 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656"] Nov 03 09:57:07.855125 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855113 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0bbca83d-8f02-412a-b508-2633ec103a94" containerName="payload-processing" Nov 03 09:57:07.855164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855128 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bbca83d-8f02-412a-b508-2633ec103a94" containerName="payload-processing" Nov 03 09:57:07.855164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855145 5015 cpu_manager.go:414] "containerMap: removing container" podUID="eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1" containerName="payload-processing" Nov 03 09:57:07.855164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855150 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1" containerName="payload-processing" Nov 03 09:57:07.855164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855157 5015 cpu_manager.go:414] "containerMap: removing container" podUID="1cc5a185-5f49-47f1-a634-65346de5e8b8" containerName="payload-pre-processing" Nov 03 09:57:07.855164 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855162 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc5a185-5f49-47f1-a634-65346de5e8b8" containerName="payload-pre-processing" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855175 5015 cpu_manager.go:414] "containerMap: removing container" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855179 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855187 5015 cpu_manager.go:414] "containerMap: removing container" podUID="190ecc81-1e36-4a1d-9b19-d1f9f2d06b15" containerName="maas-api" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855191 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190ecc81-1e36-4a1d-9b19-d1f9f2d06b15" containerName="maas-api" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855202 5015 cpu_manager.go:414] "containerMap: removing container" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855208 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855225 5015 cpu_manager.go:414] "containerMap: removing container" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855230 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855237 5015 cpu_manager.go:414] "containerMap: removing container" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855241 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855247 5015 cpu_manager.go:414] "containerMap: removing container" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855251 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855259 5015 cpu_manager.go:414] "containerMap: removing container" podUID="55a4c0d0-ad33-41df-ac33-99f23537a233" containerName="revoke-keys" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855264 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a4c0d0-ad33-41df-ac33-99f23537a233" containerName="revoke-keys" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855271 5015 cpu_manager.go:414] "containerMap: removing container" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" Nov 03 09:57:07.855321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855275 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" Nov 03 09:57:07.855804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855344 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" Nov 03 09:57:07.855804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855351 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e9349e-0cdb-4aa1-ab1b-f44b5dd0ae1f" containerName="istio-proxy" Nov 03 09:57:07.855804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855357 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="56aad1c2-84bc-40fe-bba6-a0eb6374b47a" containerName="main" Nov 03 09:57:07.855804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855363 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="448ff7d3-7b6e-49e6-9e3e-5093e589dc4d" containerName="istio-proxy" Nov 03 09:57:07.855804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855367 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a4c0d0-ad33-41df-ac33-99f23537a233" containerName="revoke-keys" Nov 03 09:57:07.855804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855373 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ab4552-a914-4ccc-a3e1-211981c0feae" containerName="istio-proxy" Nov 03 09:57:07.855804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855380 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bbca83d-8f02-412a-b508-2633ec103a94" containerName="payload-processing" Nov 03 09:57:07.855804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855386 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefad44e-6ed3-4ed4-aa75-0f1a69ea07e1" containerName="payload-processing" Nov 03 09:57:07.855804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855391 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc5a185-5f49-47f1-a634-65346de5e8b8" containerName="payload-pre-processing" Nov 03 09:57:07.855804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855398 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="190ecc81-1e36-4a1d-9b19-d1f9f2d06b15" containerName="maas-api" Nov 03 09:57:07.855804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.855404 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fafe9e0-e3d3-43d2-961a-1fd7eb9136bc" containerName="main" Nov 03 09:57:07.858703 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.858686 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:07.915337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.913511 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d37c102d-6b656\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:07.915337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.913688 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d37c102d-6b656\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:07.915337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.913786 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnmr\" (UniqueName: \"kubernetes.io/projected/4bc96401-10e3-4e01-a057-23d5c9516e1b-kube-api-access-6rnmr\") pod \"maas-api-revoke-sub-d37c102d-6b656\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:07.915337 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.915261 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656"] Nov 03 09:57:07.961944 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.961906 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 09:57:07.962128 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:07.962032 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:57:08.014871 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:08.014838 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnmr\" (UniqueName: \"kubernetes.io/projected/4bc96401-10e3-4e01-a057-23d5c9516e1b-kube-api-access-6rnmr\") pod \"maas-api-revoke-sub-d37c102d-6b656\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:08.015039 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:08.014929 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d37c102d-6b656\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:08.015039 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:08.014991 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d37c102d-6b656\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:08.015788 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:08.015712 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d37c102d-6b656\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:08.018026 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:08.018001 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d37c102d-6b656\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:08.075275 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:08.075240 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnmr\" (UniqueName: \"kubernetes.io/projected/4bc96401-10e3-4e01-a057-23d5c9516e1b-kube-api-access-6rnmr\") pod \"maas-api-revoke-sub-d37c102d-6b656\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:08.174222 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:08.174138 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:08.557633 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:57:08.557599 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc96401_10e3_4e01_a057_23d5c9516e1b.slice/crio-71d359662b47d17421284405e8bda50550911904525275e34178f733e89d27a7 WatchSource:0}: Error finding container 71d359662b47d17421284405e8bda50550911904525275e34178f733e89d27a7: Status 404 returned error can't find the container with id 71d359662b47d17421284405e8bda50550911904525275e34178f733e89d27a7 Nov 03 09:57:08.557969 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:08.557883 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656"] Nov 03 09:57:08.932299 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:08.932261 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" event={"ID":"4bc96401-10e3-4e01-a057-23d5c9516e1b","Type":"ContainerStarted","Data":"d185b714626c2a8c7e2e4a5011af6c0da0d7e1ffa518c7e71651a9b7c741debc"} Nov 03 09:57:08.932519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:08.932312 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" event={"ID":"4bc96401-10e3-4e01-a057-23d5c9516e1b","Type":"ContainerStarted","Data":"71d359662b47d17421284405e8bda50550911904525275e34178f733e89d27a7"} Nov 03 09:57:09.050180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:09.050124 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" podStartSLOduration=2.050106732 podStartE2EDuration="2.050106732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:57:09.047906597 +0000 UTC m=+4661.972848815" watchObservedRunningTime="2026-11-03 09:57:09.050106732 +0000 UTC m=+4661.975048940" Nov 03 09:57:09.516386 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:09.516348 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" podUID="1c5b9e02-d908-4819-8d82-f2ce501928a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:57:09.940548 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:09.940513 5015 generic.go:356] "Generic (PLEG): container finished" podID="4bc96401-10e3-4e01-a057-23d5c9516e1b" containerID="d185b714626c2a8c7e2e4a5011af6c0da0d7e1ffa518c7e71651a9b7c741debc" exitCode=0 Nov 03 09:57:09.940983 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:09.940596 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" event={"ID":"4bc96401-10e3-4e01-a057-23d5c9516e1b","Type":"ContainerDied","Data":"d185b714626c2a8c7e2e4a5011af6c0da0d7e1ffa518c7e71651a9b7c741debc"} Nov 03 09:57:11.103742 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.103715 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:11.254659 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.254595 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-openshift-ca\") pod \"4bc96401-10e3-4e01-a057-23d5c9516e1b\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " Nov 03 09:57:11.254874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.254773 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-xks-ca\") pod \"4bc96401-10e3-4e01-a057-23d5c9516e1b\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " Nov 03 09:57:11.254874 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.254826 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc96401-10e3-4e01-a057-23d5c9516e1b-kube-api-access-6rnmr\" (UniqueName: \"kubernetes.io/projected/4bc96401-10e3-4e01-a057-23d5c9516e1b-kube-api-access-6rnmr\") pod \"4bc96401-10e3-4e01-a057-23d5c9516e1b\" (UID: \"4bc96401-10e3-4e01-a057-23d5c9516e1b\") " Nov 03 09:57:11.255016 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.254968 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-openshift-ca" pod "4bc96401-10e3-4e01-a057-23d5c9516e1b" (UID: "4bc96401-10e3-4e01-a057-23d5c9516e1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:11.255174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.255155 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:11.257198 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.257164 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-xks-ca" pod "4bc96401-10e3-4e01-a057-23d5c9516e1b" (UID: "4bc96401-10e3-4e01-a057-23d5c9516e1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:11.257348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.257237 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc96401-10e3-4e01-a057-23d5c9516e1b-kube-api-access-6rnmr" pod "4bc96401-10e3-4e01-a057-23d5c9516e1b" (UID: "4bc96401-10e3-4e01-a057-23d5c9516e1b"). InnerVolumeSpecName "kube-api-access-6rnmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:11.356553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.356511 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc96401-10e3-4e01-a057-23d5c9516e1b-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:11.356553 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.356552 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnmr\" (UniqueName: \"kubernetes.io/projected/4bc96401-10e3-4e01-a057-23d5c9516e1b-kube-api-access-6rnmr\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:11.955587 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.955555 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" Nov 03 09:57:11.955745 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.955590 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656" event={"ID":"4bc96401-10e3-4e01-a057-23d5c9516e1b","Type":"ContainerDied","Data":"71d359662b47d17421284405e8bda50550911904525275e34178f733e89d27a7"} Nov 03 09:57:11.955745 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:11.955641 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71d359662b47d17421284405e8bda50550911904525275e34178f733e89d27a7" Nov 03 09:57:12.232139 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:12.232101 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th"] Nov 03 09:57:13.980424 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:13.980388 5015 generic.go:356] "Generic (PLEG): container finished" podID="c4df8dd8-b208-42c5-b6ef-fdc924243ba6" containerID="b490bd341aa36a451c32c48403eeb41c132f6600df744c7273eb31a9c51ca0c8" exitCode=0 Nov 03 09:57:13.980830 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:13.980450 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" event={"ID":"c4df8dd8-b208-42c5-b6ef-fdc924243ba6","Type":"ContainerDied","Data":"b490bd341aa36a451c32c48403eeb41c132f6600df744c7273eb31a9c51ca0c8"} Nov 03 09:57:14.027409 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:14.027344 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 09:57:14.202115 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:14.202079 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-kube-api-access-4dn5s\" (UniqueName: \"kubernetes.io/projected/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-kube-api-access-4dn5s\") pod \"c4df8dd8-b208-42c5-b6ef-fdc924243ba6\" (UID: \"c4df8dd8-b208-42c5-b6ef-fdc924243ba6\") " Nov 03 09:57:14.202317 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:14.202146 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-plugins-config-volume\") pod \"c4df8dd8-b208-42c5-b6ef-fdc924243ba6\" (UID: \"c4df8dd8-b208-42c5-b6ef-fdc924243ba6\") " Nov 03 09:57:14.202584 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:14.202560 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-plugins-config-volume" pod "c4df8dd8-b208-42c5-b6ef-fdc924243ba6" (UID: "c4df8dd8-b208-42c5-b6ef-fdc924243ba6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:14.202700 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:14.202679 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:14.204514 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:14.204463 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-kube-api-access-4dn5s" pod "c4df8dd8-b208-42c5-b6ef-fdc924243ba6" (UID: "c4df8dd8-b208-42c5-b6ef-fdc924243ba6"). InnerVolumeSpecName "kube-api-access-4dn5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:14.304089 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:14.303996 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dn5s\" (UniqueName: \"kubernetes.io/projected/c4df8dd8-b208-42c5-b6ef-fdc924243ba6-kube-api-access-4dn5s\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:14.992348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:14.992316 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" Nov 03 09:57:14.992348 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:14.992328 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9" event={"ID":"c4df8dd8-b208-42c5-b6ef-fdc924243ba6","Type":"ContainerDied","Data":"09d62d23fc8cdd56376cffe58c930655f8062b40ea00bedf68e30abd4f431ed2"} Nov 03 09:57:14.992837 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:14.992385 5015 scope.go:122] "RemoveContainer" containerID="b490bd341aa36a451c32c48403eeb41c132f6600df744c7273eb31a9c51ca0c8" Nov 03 09:57:15.160866 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:15.160825 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9"] Nov 03 09:57:15.211015 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:15.210980 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-77207e-6b74b98487-g6qj9"] Nov 03 09:57:16.902676 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:57:16.902627 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:57:17.206800 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:17.206727 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4df8dd8-b208-42c5-b6ef-fdc924243ba6" path="/var/lib/kubelet/pods/c4df8dd8-b208-42c5-b6ef-fdc924243ba6/volumes" Nov 03 09:57:17.494442 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:17.494406 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" podUID="494d276c-7e71-4d57-a8b7-8cfc32b6b4c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:57:17.494642 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:17.494445 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" podUID="bac4a5e6-5a35-47d9-9292-d1f1eb1c356f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:57:19.516343 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:19.516309 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" podUID="1c5b9e02-d908-4819-8d82-f2ce501928a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 09:57:21.816667 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.816635 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch"] Nov 03 09:57:21.817137 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.817123 5015 cpu_manager.go:414] "containerMap: removing container" podUID="c4df8dd8-b208-42c5-b6ef-fdc924243ba6" containerName="payload-pre-processing" Nov 03 09:57:21.817183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.817139 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4df8dd8-b208-42c5-b6ef-fdc924243ba6" containerName="payload-pre-processing" Nov 03 09:57:21.817183 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.817178 5015 cpu_manager.go:414] "containerMap: removing container" podUID="4bc96401-10e3-4e01-a057-23d5c9516e1b" containerName="revoke-keys" Nov 03 09:57:21.817249 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.817184 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc96401-10e3-4e01-a057-23d5c9516e1b" containerName="revoke-keys" Nov 03 09:57:21.817284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.817256 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4df8dd8-b208-42c5-b6ef-fdc924243ba6" containerName="payload-pre-processing" Nov 03 09:57:21.817284 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.817269 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc96401-10e3-4e01-a057-23d5c9516e1b" containerName="revoke-keys" Nov 03 09:57:21.822237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.822218 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:21.877917 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.877865 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db91d8bb-4btch\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:21.878071 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.877979 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db91d8bb-4btch\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:21.878182 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.878155 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w64j\" (UniqueName: \"kubernetes.io/projected/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-kube-api-access-7w64j\") pod \"maas-api-revoke-sub-db91d8bb-4btch\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:21.879010 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.878982 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch"] Nov 03 09:57:21.925544 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.925516 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 09:57:21.925544 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.925534 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:57:21.979301 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.979272 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w64j\" (UniqueName: \"kubernetes.io/projected/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-kube-api-access-7w64j\") pod \"maas-api-revoke-sub-db91d8bb-4btch\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:21.979449 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.979336 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db91d8bb-4btch\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:21.979449 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.979361 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db91d8bb-4btch\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:21.980039 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.980012 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db91d8bb-4btch\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:21.981910 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:21.981889 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db91d8bb-4btch\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:22.039082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:22.039056 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w64j\" (UniqueName: \"kubernetes.io/projected/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-kube-api-access-7w64j\") pod \"maas-api-revoke-sub-db91d8bb-4btch\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:22.134747 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:22.134668 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:22.516642 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:22.516615 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch"] Nov 03 09:57:22.517660 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:57:22.517635 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0082f1b9_4cf4_4ee0_9abe_a19fa84bb5bd.slice/crio-662aa0030fecd2eab29068546e6be3e4ba0b88a5c1747edb42921bc6bc57bd6f WatchSource:0}: Error finding container 662aa0030fecd2eab29068546e6be3e4ba0b88a5c1747edb42921bc6bc57bd6f: Status 404 returned error can't find the container with id 662aa0030fecd2eab29068546e6be3e4ba0b88a5c1747edb42921bc6bc57bd6f Nov 03 09:57:23.047658 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.047619 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" event={"ID":"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd","Type":"ContainerStarted","Data":"08a608d2810f326d55d467eb3d06defbf8d024857cb68b40df767578c06f5045"} Nov 03 09:57:23.048043 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.047679 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" event={"ID":"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd","Type":"ContainerStarted","Data":"662aa0030fecd2eab29068546e6be3e4ba0b88a5c1747edb42921bc6bc57bd6f"} Nov 03 09:57:23.165514 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.165408 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" podStartSLOduration=2.165393128 podStartE2EDuration="2.165393128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:57:23.163259153 +0000 UTC m=+4676.088201370" watchObservedRunningTime="2026-11-03 09:57:23.165393128 +0000 UTC m=+4676.090335342" Nov 03 09:57:23.506654 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.506626 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:57:23.548270 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.548245 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:57:23.599611 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.599579 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-kube-api-access-2vqfw\" (UniqueName: \"kubernetes.io/projected/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-kube-api-access-2vqfw\") pod \"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7\" (UID: \"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7\") " Nov 03 09:57:23.599769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.599673 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-plugins-config-volume\") pod \"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f\" (UID: \"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f\") " Nov 03 09:57:23.599769 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.599739 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-kube-api-access-6qsnz\" (UniqueName: \"kubernetes.io/projected/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-kube-api-access-6qsnz\") pod \"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f\" (UID: \"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f\") " Nov 03 09:57:23.599905 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.599790 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-plugins-config-volume\") pod \"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7\" (UID: \"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7\") " Nov 03 09:57:23.600127 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.600099 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-plugins-config-volume" pod "bac4a5e6-5a35-47d9-9292-d1f1eb1c356f" (UID: "bac4a5e6-5a35-47d9-9292-d1f1eb1c356f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:23.600237 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.600192 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-plugins-config-volume" pod "494d276c-7e71-4d57-a8b7-8cfc32b6b4c7" (UID: "494d276c-7e71-4d57-a8b7-8cfc32b6b4c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:23.601925 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.601903 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-kube-api-access-2vqfw" pod "494d276c-7e71-4d57-a8b7-8cfc32b6b4c7" (UID: "494d276c-7e71-4d57-a8b7-8cfc32b6b4c7"). InnerVolumeSpecName "kube-api-access-2vqfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:23.602047 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.602030 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-kube-api-access-6qsnz" pod "bac4a5e6-5a35-47d9-9292-d1f1eb1c356f" (UID: "bac4a5e6-5a35-47d9-9292-d1f1eb1c356f"). InnerVolumeSpecName "kube-api-access-6qsnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:23.700734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.700647 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vqfw\" (UniqueName: \"kubernetes.io/projected/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-kube-api-access-2vqfw\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:23.700734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.700676 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:23.700734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.700685 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qsnz\" (UniqueName: \"kubernetes.io/projected/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f-kube-api-access-6qsnz\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:23.700734 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:23.700695 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:24.055773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.055733 5015 generic.go:356] "Generic (PLEG): container finished" podID="0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd" containerID="08a608d2810f326d55d467eb3d06defbf8d024857cb68b40df767578c06f5045" exitCode=0 Nov 03 09:57:24.056204 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.055819 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" event={"ID":"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd","Type":"ContainerDied","Data":"08a608d2810f326d55d467eb3d06defbf8d024857cb68b40df767578c06f5045"} Nov 03 09:57:24.057270 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.057243 5015 generic.go:356] "Generic (PLEG): container finished" podID="494d276c-7e71-4d57-a8b7-8cfc32b6b4c7" containerID="79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca" exitCode=0 Nov 03 09:57:24.057403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.057286 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" Nov 03 09:57:24.057403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.057327 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" event={"ID":"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7","Type":"ContainerDied","Data":"79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca"} Nov 03 09:57:24.057403 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.057380 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5" event={"ID":"494d276c-7e71-4d57-a8b7-8cfc32b6b4c7","Type":"ContainerDied","Data":"0e11ca6579784913c054c070756640f34b185dcbda34ea58d5a4e3e0eb03d6f0"} Nov 03 09:57:24.057652 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.057414 5015 scope.go:122] "RemoveContainer" containerID="79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca" Nov 03 09:57:24.058687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.058665 5015 generic.go:356] "Generic (PLEG): container finished" podID="bac4a5e6-5a35-47d9-9292-d1f1eb1c356f" containerID="1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3" exitCode=0 Nov 03 09:57:24.058819 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.058709 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" Nov 03 09:57:24.058819 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.058749 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" event={"ID":"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f","Type":"ContainerDied","Data":"1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3"} Nov 03 09:57:24.058930 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.058839 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd" event={"ID":"bac4a5e6-5a35-47d9-9292-d1f1eb1c356f","Type":"ContainerDied","Data":"45c84b959a4aaab70bc6ab89e8dcfe469e9f88adf79d7ce62d70090128d1f2cc"} Nov 03 09:57:24.069002 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.068983 5015 scope.go:122] "RemoveContainer" containerID="79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca" Nov 03 09:57:24.069587 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:57:24.069562 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca\": container with ID starting with 79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca not found: ID does not exist" containerID="79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca" Nov 03 09:57:24.069712 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.069600 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca"} err="failed to get container status \"79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca\": rpc error: code = NotFound desc = could not find container \"79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca\": container with ID starting with 79a092a78d1fdfe19a6cbdb88782f451f82cb5e3e32dc19e0947ef545312a6ca not found: ID does not exist" Nov 03 09:57:24.069712 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.069620 5015 scope.go:122] "RemoveContainer" containerID="1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3" Nov 03 09:57:24.080646 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.080624 5015 scope.go:122] "RemoveContainer" containerID="1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3" Nov 03 09:57:24.080936 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:57:24.080912 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3\": container with ID starting with 1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3 not found: ID does not exist" containerID="1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3" Nov 03 09:57:24.081007 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.080947 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3"} err="failed to get container status \"1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3\": rpc error: code = NotFound desc = could not find container \"1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3\": container with ID starting with 1ba74766536a9a844584dd0067f81d7bf45c329cb113f9535604fc4dd77a4eb3 not found: ID does not exist" Nov 03 09:57:24.285910 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.285821 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5"] Nov 03 09:57:24.292313 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.292284 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b196c8-7987b67b75-wcqr5"] Nov 03 09:57:24.405940 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.405901 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd"] Nov 03 09:57:24.411574 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.411546 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b196c8-6dd7d7dccf-7zmrd"] Nov 03 09:57:24.821556 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.821522 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt"] Nov 03 09:57:24.821955 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.821941 5015 cpu_manager.go:414] "containerMap: removing container" podUID="bac4a5e6-5a35-47d9-9292-d1f1eb1c356f" containerName="payload-pre-processing" Nov 03 09:57:24.822001 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.821957 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac4a5e6-5a35-47d9-9292-d1f1eb1c356f" containerName="payload-pre-processing" Nov 03 09:57:24.822001 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.821978 5015 cpu_manager.go:414] "containerMap: removing container" podUID="494d276c-7e71-4d57-a8b7-8cfc32b6b4c7" containerName="payload-processing" Nov 03 09:57:24.822001 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.821983 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494d276c-7e71-4d57-a8b7-8cfc32b6b4c7" containerName="payload-processing" Nov 03 09:57:24.822106 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.822066 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac4a5e6-5a35-47d9-9292-d1f1eb1c356f" containerName="payload-pre-processing" Nov 03 09:57:24.822106 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.822076 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="494d276c-7e71-4d57-a8b7-8cfc32b6b4c7" containerName="payload-processing" Nov 03 09:57:24.827044 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.827023 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:24.881280 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.881246 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-910d50-openshift-default-dockercfg-w8xwd\"" Nov 03 09:57:24.913946 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.913908 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:24.914155 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.913949 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:24.914155 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.913982 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcn8s\" (UniqueName: \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-kube-api-access-hcn8s\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:24.914264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.914140 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:24.914264 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.914192 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:24.914336 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.914286 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:24.914336 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.914323 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:24.914429 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.914415 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:24.914466 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.914438 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:24.935062 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:24.935033 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt"] Nov 03 09:57:25.015579 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.015548 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.015579 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.015585 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.015841 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.015612 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.015841 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.015655 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.015966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.015849 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.015966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.015905 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.015966 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.015950 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.016124 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.015988 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.016124 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.016021 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcn8s\" (UniqueName: \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-kube-api-access-hcn8s\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.016124 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.016034 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.016451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.016427 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.016451 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.016443 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.016640 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.016493 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.016640 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.016634 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.018169 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.018148 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.018549 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.018532 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.075620 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.075551 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcn8s\" (UniqueName: \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-kube-api-access-hcn8s\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.075620 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.075560 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.142687 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.142651 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:25.199056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.199024 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:25.203804 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.203774 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494d276c-7e71-4d57-a8b7-8cfc32b6b4c7" path="/var/lib/kubelet/pods/494d276c-7e71-4d57-a8b7-8cfc32b6b4c7/volumes" Nov 03 09:57:25.204303 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.204282 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac4a5e6-5a35-47d9-9292-d1f1eb1c356f" path="/var/lib/kubelet/pods/bac4a5e6-5a35-47d9-9292-d1f1eb1c356f/volumes" Nov 03 09:57:25.218033 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.218004 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-openshift-ca\") pod \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " Nov 03 09:57:25.218196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.218057 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-kube-api-access-7w64j\" (UniqueName: \"kubernetes.io/projected/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-kube-api-access-7w64j\") pod \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " Nov 03 09:57:25.218196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.218083 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-xks-ca\") pod \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\" (UID: \"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd\") " Nov 03 09:57:25.218492 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.218446 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-openshift-ca" pod "0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd" (UID: "0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:25.222041 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.222012 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-xks-ca" pod "0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd" (UID: "0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:25.223449 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.223419 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-kube-api-access-7w64j" pod "0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd" (UID: "0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd"). InnerVolumeSpecName "kube-api-access-7w64j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:25.319389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.319360 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:25.319389 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.319388 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w64j\" (UniqueName: \"kubernetes.io/projected/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-kube-api-access-7w64j\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:25.319615 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.319397 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:25.582148 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.582118 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt"] Nov 03 09:57:25.582836 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:57:25.582813 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb5677e_b2a4_4c51_9537_5ccc1292ecbe.slice/crio-e61e829b61afd9de78a11744b9344dfd9546ce78ff69ceda5f3b71f6d3653dab WatchSource:0}: Error finding container e61e829b61afd9de78a11744b9344dfd9546ce78ff69ceda5f3b71f6d3653dab: Status 404 returned error can't find the container with id e61e829b61afd9de78a11744b9344dfd9546ce78ff69ceda5f3b71f6d3653dab Nov 03 09:57:25.585109 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.585079 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:57:25.585180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.585146 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:57:25.585180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.585174 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 09:57:25.876234 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.876207 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 09:57:25.925290 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.925241 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5b9e02-d908-4819-8d82-f2ce501928a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5b9e02-d908-4819-8d82-f2ce501928a7-plugins-config-volume\") pod \"1c5b9e02-d908-4819-8d82-f2ce501928a7\" (UID: \"1c5b9e02-d908-4819-8d82-f2ce501928a7\") " Nov 03 09:57:25.925527 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.925441 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5b9e02-d908-4819-8d82-f2ce501928a7-kube-api-access-p5klm\" (UniqueName: \"kubernetes.io/projected/1c5b9e02-d908-4819-8d82-f2ce501928a7-kube-api-access-p5klm\") pod \"1c5b9e02-d908-4819-8d82-f2ce501928a7\" (UID: \"1c5b9e02-d908-4819-8d82-f2ce501928a7\") " Nov 03 09:57:25.925637 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.925611 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5b9e02-d908-4819-8d82-f2ce501928a7-plugins-config-volume" pod "1c5b9e02-d908-4819-8d82-f2ce501928a7" (UID: "1c5b9e02-d908-4819-8d82-f2ce501928a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:25.925817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.925799 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5b9e02-d908-4819-8d82-f2ce501928a7-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:25.927743 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:25.927725 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5b9e02-d908-4819-8d82-f2ce501928a7-kube-api-access-p5klm" pod "1c5b9e02-d908-4819-8d82-f2ce501928a7" (UID: "1c5b9e02-d908-4819-8d82-f2ce501928a7"). InnerVolumeSpecName "kube-api-access-p5klm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:26.027181 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.027152 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5klm\" (UniqueName: \"kubernetes.io/projected/1c5b9e02-d908-4819-8d82-f2ce501928a7-kube-api-access-p5klm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:26.080636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.078380 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" event={"ID":"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe","Type":"ContainerStarted","Data":"60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f"} Nov 03 09:57:26.080636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.078453 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" event={"ID":"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe","Type":"ContainerStarted","Data":"e61e829b61afd9de78a11744b9344dfd9546ce78ff69ceda5f3b71f6d3653dab"} Nov 03 09:57:26.080636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.080428 5015 generic.go:356] "Generic (PLEG): container finished" podID="1c5b9e02-d908-4819-8d82-f2ce501928a7" containerID="1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139" exitCode=0 Nov 03 09:57:26.080636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.080526 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" Nov 03 09:57:26.080636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.080550 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" event={"ID":"1c5b9e02-d908-4819-8d82-f2ce501928a7","Type":"ContainerDied","Data":"1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139"} Nov 03 09:57:26.080636 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.080624 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l" event={"ID":"1c5b9e02-d908-4819-8d82-f2ce501928a7","Type":"ContainerDied","Data":"29c99039053bf835f3329e5eb1570876373ec926202d75dda6d5cb10297d88e5"} Nov 03 09:57:26.081541 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.080675 5015 scope.go:122] "RemoveContainer" containerID="1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139" Nov 03 09:57:26.083975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.083952 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" event={"ID":"0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd","Type":"ContainerDied","Data":"662aa0030fecd2eab29068546e6be3e4ba0b88a5c1747edb42921bc6bc57bd6f"} Nov 03 09:57:26.084107 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.083985 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch" Nov 03 09:57:26.084107 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.084000 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="662aa0030fecd2eab29068546e6be3e4ba0b88a5c1747edb42921bc6bc57bd6f" Nov 03 09:57:26.094381 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.094359 5015 scope.go:122] "RemoveContainer" containerID="1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139" Nov 03 09:57:26.094829 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:57:26.094809 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139\": container with ID starting with 1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139 not found: ID does not exist" containerID="1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139" Nov 03 09:57:26.094877 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.094842 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139"} err="failed to get container status \"1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139\": rpc error: code = NotFound desc = could not find container \"1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139\": container with ID starting with 1b381df10c203d9ca8ec16507e76db279a0de6cfb2f8f3efbc6aff2d458ce139 not found: ID does not exist" Nov 03 09:57:26.164561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.156658 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:26.164561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.158140 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 09:57:26.164561 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.158202 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 09:57:26.212360 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.212309 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" podStartSLOduration=2.212295545 podStartE2EDuration="2.212295545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:57:26.211597468 +0000 UTC m=+4679.136539708" watchObservedRunningTime="2026-11-03 09:57:26.212295545 +0000 UTC m=+4679.137237759" Nov 03 09:57:26.382024 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.381986 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l"] Nov 03 09:57:26.387299 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:26.387270 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-910d50-77d64fccd9-ctk5l"] Nov 03 09:57:27.152508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.149608 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 09:57:27.152508 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.149665 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 09:57:27.203694 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.203657 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5b9e02-d908-4819-8d82-f2ce501928a7" path="/var/lib/kubelet/pods/1c5b9e02-d908-4819-8d82-f2ce501928a7/volumes" Nov 03 09:57:27.263821 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.263779 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" containerID="cri-o://07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8" gracePeriod=45 Nov 03 09:57:27.547006 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.546663 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 09:57:27.675180 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.675143 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-home\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-home\") pod \"40f331f4-0042-4157-9107-d5626c3fc8d4\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " Nov 03 09:57:27.675406 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.675192 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f331f4-0042-4157-9107-d5626c3fc8d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/40f331f4-0042-4157-9107-d5626c3fc8d4-tls-certs\") pod \"40f331f4-0042-4157-9107-d5626c3fc8d4\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " Nov 03 09:57:27.675406 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.675323 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-model-cache\") pod \"40f331f4-0042-4157-9107-d5626c3fc8d4\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " Nov 03 09:57:27.675406 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.675378 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-dshm\") pod \"40f331f4-0042-4157-9107-d5626c3fc8d4\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " Nov 03 09:57:27.675644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.675423 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f331f4-0042-4157-9107-d5626c3fc8d4-kube-api-access-r4k82\" (UniqueName: \"kubernetes.io/projected/40f331f4-0042-4157-9107-d5626c3fc8d4-kube-api-access-r4k82\") pod \"40f331f4-0042-4157-9107-d5626c3fc8d4\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " Nov 03 09:57:27.675644 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.675547 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-tmp-dir\") pod \"40f331f4-0042-4157-9107-d5626c3fc8d4\" (UID: \"40f331f4-0042-4157-9107-d5626c3fc8d4\") " Nov 03 09:57:27.676710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.676583 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-model-cache" pod "40f331f4-0042-4157-9107-d5626c3fc8d4" (UID: "40f331f4-0042-4157-9107-d5626c3fc8d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:27.676710 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.676599 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-tmp-dir" pod "40f331f4-0042-4157-9107-d5626c3fc8d4" (UID: "40f331f4-0042-4157-9107-d5626c3fc8d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:27.676980 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.676874 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-home" pod "40f331f4-0042-4157-9107-d5626c3fc8d4" (UID: "40f331f4-0042-4157-9107-d5626c3fc8d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:27.679649 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.679540 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f331f4-0042-4157-9107-d5626c3fc8d4-kube-api-access-r4k82" pod "40f331f4-0042-4157-9107-d5626c3fc8d4" (UID: "40f331f4-0042-4157-9107-d5626c3fc8d4"). InnerVolumeSpecName "kube-api-access-r4k82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:27.679821 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.679798 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f331f4-0042-4157-9107-d5626c3fc8d4-tls-certs" pod "40f331f4-0042-4157-9107-d5626c3fc8d4" (UID: "40f331f4-0042-4157-9107-d5626c3fc8d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:27.680020 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.679984 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-dshm" pod "40f331f4-0042-4157-9107-d5626c3fc8d4" (UID: "40f331f4-0042-4157-9107-d5626c3fc8d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:27.777412 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.777376 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:27.777412 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.777409 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f331f4-0042-4157-9107-d5626c3fc8d4-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:27.777412 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.777418 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:27.777709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.777429 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:27.777709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.777439 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4k82\" (UniqueName: \"kubernetes.io/projected/40f331f4-0042-4157-9107-d5626c3fc8d4-kube-api-access-r4k82\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:27.777709 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:27.777448 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f331f4-0042-4157-9107-d5626c3fc8d4-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:28.100103 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.100065 5015 generic.go:356] "Generic (PLEG): container finished" podID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerID="07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8" exitCode=0 Nov 03 09:57:28.100316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.100108 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" Nov 03 09:57:28.100316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.100145 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" event={"ID":"40f331f4-0042-4157-9107-d5626c3fc8d4","Type":"ContainerDied","Data":"07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8"} Nov 03 09:57:28.100316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.100188 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" event={"ID":"40f331f4-0042-4157-9107-d5626c3fc8d4","Type":"ContainerDied","Data":"cbce1a06f607899183a2b23ac6ff9b289a45bc8c68961a17ee84916251ee40ef"} Nov 03 09:57:28.100316 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.100206 5015 scope.go:122] "RemoveContainer" containerID="07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8" Nov 03 09:57:28.110326 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.110303 5015 scope.go:122] "RemoveContainer" containerID="07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8" Nov 03 09:57:28.110646 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:57:28.110628 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8\": container with ID starting with 07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8 not found: ID does not exist" containerID="07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8" Nov 03 09:57:28.110724 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.110655 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8"} err="failed to get container status \"07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8\": rpc error: code = NotFound desc = could not find container \"07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8\": container with ID starting with 07181bc10f4e3f2a78f3d16e889645821b92e045d3e7b786a93e2b6c81a5b4e8 not found: ID does not exist" Nov 03 09:57:28.161619 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.161578 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 09:57:28.162123 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.161632 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 09:57:28.273979 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.273940 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th"] Nov 03 09:57:28.280882 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.280856 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th"] Nov 03 09:57:28.454458 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:28.453568 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/simulator-w3-98d5dd-kserve-676c669984-z59th" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:57:29.155904 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:29.155865 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 09:57:29.156192 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:29.156160 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 09:57:29.204018 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:29.203978 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" path="/var/lib/kubelet/pods/40f331f4-0042-4157-9107-d5626c3fc8d4/volumes" Nov 03 09:57:30.153934 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:30.153892 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 09:57:30.154247 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:30.154216 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 09:57:31.158500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:31.157869 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 09:57:31.158500 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:31.157932 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 09:57:32.143590 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:32.143546 5015 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 09:57:32.143827 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:32.143601 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 09:57:33.148220 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:33.148187 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:33.148768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:33.148594 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:33.149423 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:33.149403 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:34.147244 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:34.147205 5015 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" secret="" err="secret \"e2e-worker-w0-910d50-openshift-default-dockercfg-w8xwd\" not found" Nov 03 09:57:34.197146 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:34.197113 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt"] Nov 03 09:57:35.152433 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:35.152398 5015 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" secret="" err="secret \"e2e-worker-w0-910d50-openshift-default-dockercfg-w8xwd\" not found" Nov 03 09:57:36.158396 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:36.158347 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerName="istio-proxy" containerID="cri-o://60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f" gracePeriod=30 Nov 03 09:57:41.412892 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.412866 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:41.523588 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.523549 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-token\" (UniqueName: \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-token\") pod \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " Nov 03 09:57:41.523752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.523646 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-podinfo\") pod \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " Nov 03 09:57:41.523752 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.523694 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-kube-api-access-hcn8s\" (UniqueName: \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-kube-api-access-hcn8s\") pod \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " Nov 03 09:57:41.523899 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.523748 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-socket\") pod \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " Nov 03 09:57:41.523899 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.523790 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-envoy\") pod \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " Nov 03 09:57:41.523899 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.523834 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-data\") pod \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " Nov 03 09:57:41.523899 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.523884 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istiod-ca-cert\") pod \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " Nov 03 09:57:41.524101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.523922 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-certs\") pod \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " Nov 03 09:57:41.524101 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.523956 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-credential-socket\") pod \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\" (UID: \"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe\") " Nov 03 09:57:41.524223 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.524111 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-socket" pod "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" (UID: "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:41.524223 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.524132 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-data" pod "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" (UID: "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:41.524345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.524300 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-certs" pod "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" (UID: "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:41.524345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.524321 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:41.524345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.524335 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-credential-socket" pod "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" (UID: "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:41.524545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.524345 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:41.524545 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.524370 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istiod-ca-cert" pod "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" (UID: "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:41.526141 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.526112 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-podinfo" pod "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" (UID: "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:57:41.526519 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.526500 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-kube-api-access-hcn8s" pod "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" (UID: "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe"). InnerVolumeSpecName "kube-api-access-hcn8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:41.526653 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.526633 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-envoy" pod "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" (UID: "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:41.526714 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.526648 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-token" pod "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" (UID: "3bb5677e-b2a4-4c51-9537-5ccc1292ecbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:41.625330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.625293 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:41.625330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.625323 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:41.625330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.625332 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:41.625330 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.625342 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:41.625677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.625351 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:41.625677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.625359 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:41.625677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:41.625368 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcn8s\" (UniqueName: \"kubernetes.io/projected/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe-kube-api-access-hcn8s\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:42.194599 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:42.194561 5015 generic.go:356] "Generic (PLEG): container finished" podID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerID="60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f" exitCode=0 Nov 03 09:57:42.194817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:42.194620 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" Nov 03 09:57:42.194817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:42.194652 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" event={"ID":"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe","Type":"ContainerDied","Data":"60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f"} Nov 03 09:57:42.194817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:42.194716 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt" event={"ID":"3bb5677e-b2a4-4c51-9537-5ccc1292ecbe","Type":"ContainerDied","Data":"e61e829b61afd9de78a11744b9344dfd9546ce78ff69ceda5f3b71f6d3653dab"} Nov 03 09:57:42.194817 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:42.194739 5015 scope.go:122] "RemoveContainer" containerID="60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f" Nov 03 09:57:42.205194 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:42.205175 5015 scope.go:122] "RemoveContainer" containerID="60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f" Nov 03 09:57:42.205524 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:57:42.205468 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f\": container with ID starting with 60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f not found: ID does not exist" containerID="60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f" Nov 03 09:57:42.205589 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:42.205536 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f"} err="failed to get container status \"60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f\": rpc error: code = NotFound desc = could not find container \"60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f\": container with ID starting with 60d722a002bc66905fbffbbe6a44fc9a605567490df6aa6a61db9912a4a8285f not found: ID does not exist" Nov 03 09:57:42.413792 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:42.413746 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt"] Nov 03 09:57:42.464230 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:42.464142 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-lhzqt"] Nov 03 09:57:43.203681 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:43.203650 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" path="/var/lib/kubelet/pods/3bb5677e-b2a4-4c51-9537-5ccc1292ecbe/volumes" Nov 03 09:57:45.770826 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:45.770792 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7"] Nov 03 09:57:45.771333 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:45.771045 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" containerID="cri-o://38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177" gracePeriod=30 Nov 03 09:57:46.023286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:46.023209 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:57:46.023286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:46.023265 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:57:46.901981 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:57:46.901945 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:57:51.029243 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.029217 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:57:51.121939 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.121861 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-envoy\") pod \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " Nov 03 09:57:51.121939 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.121895 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-socket\") pod \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " Nov 03 09:57:51.121939 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.121925 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istiod-ca-cert\") pod \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " Nov 03 09:57:51.122227 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122054 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-podinfo\") pod \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " Nov 03 09:57:51.122227 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122085 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-data\") pod \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " Nov 03 09:57:51.122227 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122125 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-token\" (UniqueName: \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-token\") pod \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " Nov 03 09:57:51.122227 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122180 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-certs\") pod \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " Nov 03 09:57:51.122227 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122195 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-socket" pod "e84b7f8b-0f44-4c2c-86e9-9198d73399f3" (UID: "e84b7f8b-0f44-4c2c-86e9-9198d73399f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:51.122533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122215 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-credential-socket\") pod \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " Nov 03 09:57:51.122533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122280 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-kube-api-access-hxgtj\" (UniqueName: \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-kube-api-access-hxgtj\") pod \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\" (UID: \"e84b7f8b-0f44-4c2c-86e9-9198d73399f3\") " Nov 03 09:57:51.122533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122426 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-data" pod "e84b7f8b-0f44-4c2c-86e9-9198d73399f3" (UID: "e84b7f8b-0f44-4c2c-86e9-9198d73399f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:51.122533 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122446 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istiod-ca-cert" pod "e84b7f8b-0f44-4c2c-86e9-9198d73399f3" (UID: "e84b7f8b-0f44-4c2c-86e9-9198d73399f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:51.122792 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122563 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-certs" pod "e84b7f8b-0f44-4c2c-86e9-9198d73399f3" (UID: "e84b7f8b-0f44-4c2c-86e9-9198d73399f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:51.122792 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122617 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-credential-socket" pod "e84b7f8b-0f44-4c2c-86e9-9198d73399f3" (UID: "e84b7f8b-0f44-4c2c-86e9-9198d73399f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:51.122884 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122872 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:51.122929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122892 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:51.122929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122908 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:51.122929 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122921 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:51.123103 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.122937 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:51.124378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.124347 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-podinfo" pod "e84b7f8b-0f44-4c2c-86e9-9198d73399f3" (UID: "e84b7f8b-0f44-4c2c-86e9-9198d73399f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:57:51.124378 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.124373 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-envoy" pod "e84b7f8b-0f44-4c2c-86e9-9198d73399f3" (UID: "e84b7f8b-0f44-4c2c-86e9-9198d73399f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:51.124635 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.124620 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-token" pod "e84b7f8b-0f44-4c2c-86e9-9198d73399f3" (UID: "e84b7f8b-0f44-4c2c-86e9-9198d73399f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:51.124964 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.124951 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-kube-api-access-hxgtj" pod "e84b7f8b-0f44-4c2c-86e9-9198d73399f3" (UID: "e84b7f8b-0f44-4c2c-86e9-9198d73399f3"). InnerVolumeSpecName "kube-api-access-hxgtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:51.223716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.223682 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:51.223716 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.223713 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:51.223951 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.223723 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxgtj\" (UniqueName: \"kubernetes.io/projected/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-kube-api-access-hxgtj\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:51.223951 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.223734 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84b7f8b-0f44-4c2c-86e9-9198d73399f3-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:51.252540 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.252503 5015 generic.go:356] "Generic (PLEG): container finished" podID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerID="38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177" exitCode=0 Nov 03 09:57:51.252715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.252548 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" event={"ID":"e84b7f8b-0f44-4c2c-86e9-9198d73399f3","Type":"ContainerDied","Data":"38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177"} Nov 03 09:57:51.252715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.252572 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" Nov 03 09:57:51.252715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.252618 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7" event={"ID":"e84b7f8b-0f44-4c2c-86e9-9198d73399f3","Type":"ContainerDied","Data":"c7b3bce9df64f202a1233e789d6db53e6da8d529992db600be908635565bab59"} Nov 03 09:57:51.252715 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.252651 5015 scope.go:122] "RemoveContainer" containerID="38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177" Nov 03 09:57:51.264959 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.264937 5015 scope.go:122] "RemoveContainer" containerID="38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177" Nov 03 09:57:51.265220 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:57:51.265196 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177\": container with ID starting with 38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177 not found: ID does not exist" containerID="38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177" Nov 03 09:57:51.265332 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.265227 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177"} err="failed to get container status \"38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177\": rpc error: code = NotFound desc = could not find container \"38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177\": container with ID starting with 38d6014ae108754dd6286b9d6e4c0dcba442a30a9c053c7cd4ff2b3ee1d14177 not found: ID does not exist" Nov 03 09:57:51.421836 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.421746 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7"] Nov 03 09:57:51.427457 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:51.427430 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d6bq7"] Nov 03 09:57:53.203939 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:53.203895 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" path="/var/lib/kubelet/pods/e84b7f8b-0f44-4c2c-86e9-9198d73399f3/volumes" Nov 03 09:57:54.744471 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:54.744435 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dfcc7c588-7hvqz_ae051d13-de71-4752-b918-8bb60b21002d/authorino/0.log" Nov 03 09:57:55.116976 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:55.116886 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rc42m_1786e9e7-2202-4fce-aaf4-2f0b30932688/manager/0.log" Nov 03 09:57:56.961510 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:56.961448 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8bqv8_05e4fc7a-c29d-4829-84f6-06cd99b8dae1/manager/0.log" Nov 03 09:57:59.304173 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:57:59.304136 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wwbxd_e6a48fda-d04e-423f-89dc-a640c4b2d513/manager/0.log" Nov 03 09:58:16.902171 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:58:16.902085 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:58:23.364345 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.364314 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj"] Nov 03 09:58:23.364907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.364809 5015 cpu_manager.go:414] "containerMap: removing container" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" Nov 03 09:58:23.364907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.364828 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" Nov 03 09:58:23.364907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.364840 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd" containerName="revoke-keys" Nov 03 09:58:23.364907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.364847 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd" containerName="revoke-keys" Nov 03 09:58:23.364907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.364867 5015 cpu_manager.go:414] "containerMap: removing container" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerName="istio-proxy" Nov 03 09:58:23.364907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.364874 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerName="istio-proxy" Nov 03 09:58:23.364907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.364887 5015 cpu_manager.go:414] "containerMap: removing container" podUID="1c5b9e02-d908-4819-8d82-f2ce501928a7" containerName="payload-processing" Nov 03 09:58:23.364907 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.364894 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5b9e02-d908-4819-8d82-f2ce501928a7" containerName="payload-processing" Nov 03 09:58:23.365179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.364921 5015 cpu_manager.go:414] "containerMap: removing container" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" Nov 03 09:58:23.365179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.364929 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" Nov 03 09:58:23.365179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.365031 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f331f4-0042-4157-9107-d5626c3fc8d4" containerName="main" Nov 03 09:58:23.365179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.365039 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd" containerName="revoke-keys" Nov 03 09:58:23.365179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.365048 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb5677e-b2a4-4c51-9537-5ccc1292ecbe" containerName="istio-proxy" Nov 03 09:58:23.365179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.365054 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5b9e02-d908-4819-8d82-f2ce501928a7" containerName="payload-processing" Nov 03 09:58:23.365179 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.365062 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84b7f8b-0f44-4c2c-86e9-9198d73399f3" containerName="istio-proxy" Nov 03 09:58:23.368166 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.368145 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:58:23.425694 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.425658 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd0b3fd-d568-4095-81d4-1610fce17797-plugins-config-volume\") pod \"payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj\" (UID: \"4cd0b3fd-d568-4095-81d4-1610fce17797\") " pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:58:23.425886 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.425855 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbfgx\" (UniqueName: \"kubernetes.io/projected/4cd0b3fd-d568-4095-81d4-1610fce17797-kube-api-access-jbfgx\") pod \"payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj\" (UID: \"4cd0b3fd-d568-4095-81d4-1610fce17797\") " pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:58:23.426114 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.426096 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj"] Nov 03 09:58:23.473380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.473355 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a3965d89-dockercfg-gwhfc\"" Nov 03 09:58:23.473380 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.473351 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a3965d89\"" Nov 03 09:58:23.526663 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.526628 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbfgx\" (UniqueName: \"kubernetes.io/projected/4cd0b3fd-d568-4095-81d4-1610fce17797-kube-api-access-jbfgx\") pod \"payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj\" (UID: \"4cd0b3fd-d568-4095-81d4-1610fce17797\") " pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:58:23.526993 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.526686 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd0b3fd-d568-4095-81d4-1610fce17797-plugins-config-volume\") pod \"payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj\" (UID: \"4cd0b3fd-d568-4095-81d4-1610fce17797\") " pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:58:23.527614 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.527592 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd0b3fd-d568-4095-81d4-1610fce17797-plugins-config-volume\") pod \"payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj\" (UID: \"4cd0b3fd-d568-4095-81d4-1610fce17797\") " pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:58:23.542042 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.542005 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb"] Nov 03 09:58:23.545713 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.545692 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:58:23.586142 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.586117 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbfgx\" (UniqueName: \"kubernetes.io/projected/4cd0b3fd-d568-4095-81d4-1610fce17797-kube-api-access-jbfgx\") pod \"payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj\" (UID: \"4cd0b3fd-d568-4095-81d4-1610fce17797\") " pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:58:23.601945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.601909 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb"] Nov 03 09:58:23.627899 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.627836 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7kt9\" (UniqueName: \"kubernetes.io/projected/2f55d3dd-082f-4161-9330-424b3e27907f-kube-api-access-x7kt9\") pod \"payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb\" (UID: \"2f55d3dd-082f-4161-9330-424b3e27907f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:58:23.628042 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.628017 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f55d3dd-082f-4161-9330-424b3e27907f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb\" (UID: \"2f55d3dd-082f-4161-9330-424b3e27907f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:58:23.683108 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.683082 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:58:23.728823 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.728787 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f55d3dd-082f-4161-9330-424b3e27907f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb\" (UID: \"2f55d3dd-082f-4161-9330-424b3e27907f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:58:23.728992 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.728834 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7kt9\" (UniqueName: \"kubernetes.io/projected/2f55d3dd-082f-4161-9330-424b3e27907f-kube-api-access-x7kt9\") pod \"payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb\" (UID: \"2f55d3dd-082f-4161-9330-424b3e27907f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:58:23.729513 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.729466 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f55d3dd-082f-4161-9330-424b3e27907f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb\" (UID: \"2f55d3dd-082f-4161-9330-424b3e27907f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:58:23.788100 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.788060 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7kt9\" (UniqueName: \"kubernetes.io/projected/2f55d3dd-082f-4161-9330-424b3e27907f-kube-api-access-x7kt9\") pod \"payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb\" (UID: \"2f55d3dd-082f-4161-9330-424b3e27907f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:58:23.858556 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:23.858521 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:58:24.056975 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:24.056945 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj"] Nov 03 09:58:24.058380 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:58:24.058344 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd0b3fd_d568_4095_81d4_1610fce17797.slice/crio-0d0eb679bb31054059ad32d789cd871478ce58dfe240f6ff93225374ed435fb1 WatchSource:0}: Error finding container 0d0eb679bb31054059ad32d789cd871478ce58dfe240f6ff93225374ed435fb1: Status 404 returned error can't find the container with id 0d0eb679bb31054059ad32d789cd871478ce58dfe240f6ff93225374ed435fb1 Nov 03 09:58:24.235105 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:24.235080 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb"] Nov 03 09:58:24.236266 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:58:24.236235 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f55d3dd_082f_4161_9330_424b3e27907f.slice/crio-16c9c26883ef2bab402dcc8fd24d2004a6c73419e47c340fefef5268066f2302 WatchSource:0}: Error finding container 16c9c26883ef2bab402dcc8fd24d2004a6c73419e47c340fefef5268066f2302: Status 404 returned error can't find the container with id 16c9c26883ef2bab402dcc8fd24d2004a6c73419e47c340fefef5268066f2302 Nov 03 09:58:24.449864 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:24.449824 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" event={"ID":"2f55d3dd-082f-4161-9330-424b3e27907f","Type":"ContainerStarted","Data":"de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0"} Nov 03 09:58:24.450342 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:24.449890 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" event={"ID":"2f55d3dd-082f-4161-9330-424b3e27907f","Type":"ContainerStarted","Data":"16c9c26883ef2bab402dcc8fd24d2004a6c73419e47c340fefef5268066f2302"} Nov 03 09:58:24.450342 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:24.449934 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:58:24.451529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:24.451496 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" event={"ID":"4cd0b3fd-d568-4095-81d4-1610fce17797","Type":"ContainerStarted","Data":"5b2d604fb9a03a67cf7720fbe0ab9664007546230bf333c7247cc949aafc6fb9"} Nov 03 09:58:24.451678 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:24.451543 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" event={"ID":"4cd0b3fd-d568-4095-81d4-1610fce17797","Type":"ContainerStarted","Data":"0d0eb679bb31054059ad32d789cd871478ce58dfe240f6ff93225374ed435fb1"} Nov 03 09:58:24.451678 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:24.451568 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:58:24.567555 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:24.567442 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" podStartSLOduration=1.5674287580000001 podStartE2EDuration="1.567428758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:58:24.565036278 +0000 UTC m=+4737.489978497" watchObservedRunningTime="2026-11-03 09:58:24.567428758 +0000 UTC m=+4737.492371008" Nov 03 09:58:24.682841 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:24.682781 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" podStartSLOduration=1.682760537 podStartE2EDuration="1.682760537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:58:24.680725298 +0000 UTC m=+4737.605667533" watchObservedRunningTime="2026-11-03 09:58:24.682760537 +0000 UTC m=+4737.607702747" Nov 03 09:58:35.461328 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:35.461292 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:58:35.461787 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:35.461375 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:58:46.901465 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:58:46.901425 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:58:56.014492 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.014451 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb"] Nov 03 09:58:56.018286 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.018265 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:56.025287 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.025044 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a3965d89-clppb\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:56.025287 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.025161 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a3965d89-clppb\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:56.025287 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.025215 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjth4\" (UniqueName: \"kubernetes.io/projected/caae089b-89b1-41fb-b91e-36988f2ee61e-kube-api-access-xjth4\") pod \"maas-api-revoke-keys-e2e-ait-a3965d89-clppb\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:56.076242 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.076213 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb"] Nov 03 09:58:56.122945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.122909 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 09:58:56.122945 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.122924 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 09:58:56.125824 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.125802 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a3965d89-clppb\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:56.125899 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.125844 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a3965d89-clppb\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:56.125899 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.125872 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjth4\" (UniqueName: \"kubernetes.io/projected/caae089b-89b1-41fb-b91e-36988f2ee61e-kube-api-access-xjth4\") pod \"maas-api-revoke-keys-e2e-ait-a3965d89-clppb\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:56.126463 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.126443 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a3965d89-clppb\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:56.128405 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.128383 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a3965d89-clppb\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:56.185135 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.185106 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjth4\" (UniqueName: \"kubernetes.io/projected/caae089b-89b1-41fb-b91e-36988f2ee61e-kube-api-access-xjth4\") pod \"maas-api-revoke-keys-e2e-ait-a3965d89-clppb\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:56.331239 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.331163 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:56.705888 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:56.705864 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb"] Nov 03 09:58:56.707703 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:58:56.707676 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaae089b_89b1_41fb_b91e_36988f2ee61e.slice/crio-7955fa030562c6002b326976ed419564562b65d10f78851b2894c2efb1568389 WatchSource:0}: Error finding container 7955fa030562c6002b326976ed419564562b65d10f78851b2894c2efb1568389: Status 404 returned error can't find the container with id 7955fa030562c6002b326976ed419564562b65d10f78851b2894c2efb1568389 Nov 03 09:58:57.656656 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:57.656623 5015 generic.go:356] "Generic (PLEG): container finished" podID="caae089b-89b1-41fb-b91e-36988f2ee61e" containerID="3b5971a04fa237c2cd31507d421c399ded5b3969b99de3baba48bb4b6817d455" exitCode=0 Nov 03 09:58:57.657056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:57.656712 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" event={"ID":"caae089b-89b1-41fb-b91e-36988f2ee61e","Type":"ContainerDied","Data":"3b5971a04fa237c2cd31507d421c399ded5b3969b99de3baba48bb4b6817d455"} Nov 03 09:58:57.657056 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:57.656764 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" event={"ID":"caae089b-89b1-41fb-b91e-36988f2ee61e","Type":"ContainerStarted","Data":"7955fa030562c6002b326976ed419564562b65d10f78851b2894c2efb1568389"} Nov 03 09:58:58.799856 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:58.799829 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:58.847423 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:58.847394 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caae089b-89b1-41fb-b91e-36988f2ee61e-kube-api-access-xjth4\" (UniqueName: \"kubernetes.io/projected/caae089b-89b1-41fb-b91e-36988f2ee61e-kube-api-access-xjth4\") pod \"caae089b-89b1-41fb-b91e-36988f2ee61e\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " Nov 03 09:58:58.849601 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:58.849576 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caae089b-89b1-41fb-b91e-36988f2ee61e-kube-api-access-xjth4" pod "caae089b-89b1-41fb-b91e-36988f2ee61e" (UID: "caae089b-89b1-41fb-b91e-36988f2ee61e"). InnerVolumeSpecName "kube-api-access-xjth4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:58:58.948445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:58.948380 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-xks-ca\") pod \"caae089b-89b1-41fb-b91e-36988f2ee61e\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " Nov 03 09:58:58.948445 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:58.948417 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-openshift-ca\") pod \"caae089b-89b1-41fb-b91e-36988f2ee61e\" (UID: \"caae089b-89b1-41fb-b91e-36988f2ee61e\") " Nov 03 09:58:58.948762 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:58.948743 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjth4\" (UniqueName: \"kubernetes.io/projected/caae089b-89b1-41fb-b91e-36988f2ee61e-kube-api-access-xjth4\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:58:58.948848 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:58.948825 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-openshift-ca" pod "caae089b-89b1-41fb-b91e-36988f2ee61e" (UID: "caae089b-89b1-41fb-b91e-36988f2ee61e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:58:58.951133 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:58.951110 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-xks-ca" pod "caae089b-89b1-41fb-b91e-36988f2ee61e" (UID: "caae089b-89b1-41fb-b91e-36988f2ee61e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:58:59.049684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:59.049658 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:58:59.049684 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:59.049682 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caae089b-89b1-41fb-b91e-36988f2ee61e-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:58:59.669671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:59.669641 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" Nov 03 09:58:59.669671 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:59.669656 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb" event={"ID":"caae089b-89b1-41fb-b91e-36988f2ee61e","Type":"ContainerDied","Data":"7955fa030562c6002b326976ed419564562b65d10f78851b2894c2efb1568389"} Nov 03 09:58:59.669936 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:58:59.669699 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7955fa030562c6002b326976ed419564562b65d10f78851b2894c2efb1568389" Nov 03 09:59:07.535526 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:07.535467 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj"] Nov 03 09:59:07.643881 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:07.643850 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb"] Nov 03 09:59:12.536768 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.536731 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" podUID="4cd0b3fd-d568-4095-81d4-1610fce17797" containerName="payload-processing" containerID="cri-o://5b2d604fb9a03a67cf7720fbe0ab9664007546230bf333c7247cc949aafc6fb9" gracePeriod=25 Nov 03 09:59:12.644729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.644616 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" podUID="2f55d3dd-082f-4161-9330-424b3e27907f" containerName="payload-pre-processing" containerID="cri-o://de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0" gracePeriod=25 Nov 03 09:59:12.746779 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.746744 5015 generic.go:356] "Generic (PLEG): container finished" podID="4cd0b3fd-d568-4095-81d4-1610fce17797" containerID="5b2d604fb9a03a67cf7720fbe0ab9664007546230bf333c7247cc949aafc6fb9" exitCode=0 Nov 03 09:59:12.746953 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.746816 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" event={"ID":"4cd0b3fd-d568-4095-81d4-1610fce17797","Type":"ContainerDied","Data":"5b2d604fb9a03a67cf7720fbe0ab9664007546230bf333c7247cc949aafc6fb9"} Nov 03 09:59:12.791061 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.790989 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:59:12.870885 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.870857 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd0b3fd-d568-4095-81d4-1610fce17797-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd0b3fd-d568-4095-81d4-1610fce17797-plugins-config-volume\") pod \"4cd0b3fd-d568-4095-81d4-1610fce17797\" (UID: \"4cd0b3fd-d568-4095-81d4-1610fce17797\") " Nov 03 09:59:12.871081 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.871031 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd0b3fd-d568-4095-81d4-1610fce17797-kube-api-access-jbfgx\" (UniqueName: \"kubernetes.io/projected/4cd0b3fd-d568-4095-81d4-1610fce17797-kube-api-access-jbfgx\") pod \"4cd0b3fd-d568-4095-81d4-1610fce17797\" (UID: \"4cd0b3fd-d568-4095-81d4-1610fce17797\") " Nov 03 09:59:12.871293 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.871267 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd0b3fd-d568-4095-81d4-1610fce17797-plugins-config-volume" pod "4cd0b3fd-d568-4095-81d4-1610fce17797" (UID: "4cd0b3fd-d568-4095-81d4-1610fce17797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:59:12.873312 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.873279 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd0b3fd-d568-4095-81d4-1610fce17797-kube-api-access-jbfgx" pod "4cd0b3fd-d568-4095-81d4-1610fce17797" (UID: "4cd0b3fd-d568-4095-81d4-1610fce17797"). InnerVolumeSpecName "kube-api-access-jbfgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:59:12.891529 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.891510 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:59:12.972528 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.972499 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f55d3dd-082f-4161-9330-424b3e27907f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f55d3dd-082f-4161-9330-424b3e27907f-plugins-config-volume\") pod \"2f55d3dd-082f-4161-9330-424b3e27907f\" (UID: \"2f55d3dd-082f-4161-9330-424b3e27907f\") " Nov 03 09:59:12.972702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.972601 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f55d3dd-082f-4161-9330-424b3e27907f-kube-api-access-x7kt9\" (UniqueName: \"kubernetes.io/projected/2f55d3dd-082f-4161-9330-424b3e27907f-kube-api-access-x7kt9\") pod \"2f55d3dd-082f-4161-9330-424b3e27907f\" (UID: \"2f55d3dd-082f-4161-9330-424b3e27907f\") " Nov 03 09:59:12.972799 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.972772 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd0b3fd-d568-4095-81d4-1610fce17797-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:12.972865 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.972801 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbfgx\" (UniqueName: \"kubernetes.io/projected/4cd0b3fd-d568-4095-81d4-1610fce17797-kube-api-access-jbfgx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:12.972924 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.972863 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f55d3dd-082f-4161-9330-424b3e27907f-plugins-config-volume" pod "2f55d3dd-082f-4161-9330-424b3e27907f" (UID: "2f55d3dd-082f-4161-9330-424b3e27907f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:59:12.974910 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:12.974886 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f55d3dd-082f-4161-9330-424b3e27907f-kube-api-access-x7kt9" pod "2f55d3dd-082f-4161-9330-424b3e27907f" (UID: "2f55d3dd-082f-4161-9330-424b3e27907f"). InnerVolumeSpecName "kube-api-access-x7kt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:59:13.073578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.073508 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7kt9\" (UniqueName: \"kubernetes.io/projected/2f55d3dd-082f-4161-9330-424b3e27907f-kube-api-access-x7kt9\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:13.073578 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.073533 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f55d3dd-082f-4161-9330-424b3e27907f-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:13.754133 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.754091 5015 generic.go:356] "Generic (PLEG): container finished" podID="2f55d3dd-082f-4161-9330-424b3e27907f" containerID="de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0" exitCode=0 Nov 03 09:59:13.754677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.754176 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" Nov 03 09:59:13.754677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.754196 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" event={"ID":"2f55d3dd-082f-4161-9330-424b3e27907f","Type":"ContainerDied","Data":"de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0"} Nov 03 09:59:13.754677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.754258 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb" event={"ID":"2f55d3dd-082f-4161-9330-424b3e27907f","Type":"ContainerDied","Data":"16c9c26883ef2bab402dcc8fd24d2004a6c73419e47c340fefef5268066f2302"} Nov 03 09:59:13.754677 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.754290 5015 scope.go:122] "RemoveContainer" containerID="de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0" Nov 03 09:59:13.755782 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.755748 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" event={"ID":"4cd0b3fd-d568-4095-81d4-1610fce17797","Type":"ContainerDied","Data":"0d0eb679bb31054059ad32d789cd871478ce58dfe240f6ff93225374ed435fb1"} Nov 03 09:59:13.755897 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.755802 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj" Nov 03 09:59:13.764332 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.764306 5015 scope.go:122] "RemoveContainer" containerID="de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0" Nov 03 09:59:13.764635 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:59:13.764616 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0\": container with ID starting with de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0 not found: ID does not exist" containerID="de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0" Nov 03 09:59:13.764702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.764645 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0"} err="failed to get container status \"de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0\": rpc error: code = NotFound desc = could not find container \"de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0\": container with ID starting with de89c1fe22379c25e752044eb8d7d78c53aaf2bec941f54a9f894735139e47b0 not found: ID does not exist" Nov 03 09:59:13.764702 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.764662 5015 scope.go:122] "RemoveContainer" containerID="5b2d604fb9a03a67cf7720fbe0ab9664007546230bf333c7247cc949aafc6fb9" Nov 03 09:59:13.923321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.923241 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj"] Nov 03 09:59:13.973954 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:13.973923 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a3965d89-75674847cc-6x4mj"] Nov 03 09:59:14.039932 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:14.039900 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb"] Nov 03 09:59:14.045379 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:14.045354 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a3965d89-7b48dbf8d7-2xjzb"] Nov 03 09:59:15.202535 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:15.202505 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f55d3dd-082f-4161-9330-424b3e27907f" path="/var/lib/kubelet/pods/2f55d3dd-082f-4161-9330-424b3e27907f/volumes" Nov 03 09:59:15.202928 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:15.202897 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd0b3fd-d568-4095-81d4-1610fce17797" path="/var/lib/kubelet/pods/4cd0b3fd-d568-4095-81d4-1610fce17797/volumes" Nov 03 09:59:16.901425 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:59:16.901393 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:59:18.421429 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:18.421392 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb"] Nov 03 09:59:18.471490 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:18.471447 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a3965d89-clppb"] Nov 03 09:59:19.202799 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:19.202763 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caae089b-89b1-41fb-b91e-36988f2ee61e" path="/var/lib/kubelet/pods/caae089b-89b1-41fb-b91e-36988f2ee61e/volumes" Nov 03 09:59:28.107082 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:28.107056 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:59:28.137321 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:28.137296 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 09:59:46.901181 ip-10-0-131-14 kubenswrapper[5015]: E1103 09:59:46.901089 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 09:59:51.449729 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.449686 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs"] Nov 03 09:59:51.450114 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.450099 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2f55d3dd-082f-4161-9330-424b3e27907f" containerName="payload-pre-processing" Nov 03 09:59:51.450114 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.450109 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f55d3dd-082f-4161-9330-424b3e27907f" containerName="payload-pre-processing" Nov 03 09:59:51.450196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.450122 5015 cpu_manager.go:414] "containerMap: removing container" podUID="caae089b-89b1-41fb-b91e-36988f2ee61e" containerName="revoke-keys" Nov 03 09:59:51.450196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.450127 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caae089b-89b1-41fb-b91e-36988f2ee61e" containerName="revoke-keys" Nov 03 09:59:51.450196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.450142 5015 cpu_manager.go:414] "containerMap: removing container" podUID="4cd0b3fd-d568-4095-81d4-1610fce17797" containerName="payload-processing" Nov 03 09:59:51.450196 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.450146 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd0b3fd-d568-4095-81d4-1610fce17797" containerName="payload-processing" Nov 03 09:59:51.450327 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.450233 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f55d3dd-082f-4161-9330-424b3e27907f" containerName="payload-pre-processing" Nov 03 09:59:51.450327 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.450241 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="caae089b-89b1-41fb-b91e-36988f2ee61e" containerName="revoke-keys" Nov 03 09:59:51.450327 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.450257 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd0b3fd-d568-4095-81d4-1610fce17797" containerName="payload-processing" Nov 03 09:59:51.452809 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.452793 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:51.511120 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.511093 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs"] Nov 03 09:59:51.531210 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.531179 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rr4bs\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:51.531351 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.531288 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvbx\" (UniqueName: \"kubernetes.io/projected/92a3f6eb-9737-4c55-a115-a88d1a8199ad-kube-api-access-wcvbx\") pod \"maas-api-revoke-sub-85237e45-rr4bs\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:51.531351 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.531329 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rr4bs\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:51.557424 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.557400 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 09:59:51.557585 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.557517 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 09:59:51.632319 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.632285 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rr4bs\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:51.632509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.632380 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rr4bs\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:51.632509 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.632413 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvbx\" (UniqueName: \"kubernetes.io/projected/92a3f6eb-9737-4c55-a115-a88d1a8199ad-kube-api-access-wcvbx\") pod \"maas-api-revoke-sub-85237e45-rr4bs\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:51.640447 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.633642 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rr4bs\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:51.642623 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.642587 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rr4bs\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:51.692294 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.692267 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvbx\" (UniqueName: \"kubernetes.io/projected/92a3f6eb-9737-4c55-a115-a88d1a8199ad-kube-api-access-wcvbx\") pod \"maas-api-revoke-sub-85237e45-rr4bs\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:51.766174 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:51.766147 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:52.141138 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:52.141104 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs"] Nov 03 09:59:52.142161 ip-10-0-131-14 kubenswrapper[5015]: W1103 09:59:52.142130 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a3f6eb_9737_4c55_a115_a88d1a8199ad.slice/crio-aff93d985160b5cf73a24029952a32bae3c2761884d7482b7f7018672f3784be WatchSource:0}: Error finding container aff93d985160b5cf73a24029952a32bae3c2761884d7482b7f7018672f3784be: Status 404 returned error can't find the container with id aff93d985160b5cf73a24029952a32bae3c2761884d7482b7f7018672f3784be Nov 03 09:59:52.983505 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:52.983440 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" event={"ID":"92a3f6eb-9737-4c55-a115-a88d1a8199ad","Type":"ContainerStarted","Data":"15d6c8524929d7e57917a1dcefbc1cbdd57a9565db058447eedcea2ef5fb0999"} Nov 03 09:59:52.983919 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:52.983559 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" event={"ID":"92a3f6eb-9737-4c55-a115-a88d1a8199ad","Type":"ContainerStarted","Data":"aff93d985160b5cf73a24029952a32bae3c2761884d7482b7f7018672f3784be"} Nov 03 09:59:53.098665 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:53.098603 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" podStartSLOduration=2.098588144 podStartE2EDuration="2.098588144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:59:53.098356483 +0000 UTC m=+4826.023298724" watchObservedRunningTime="2026-11-03 09:59:53.098588144 +0000 UTC m=+4826.023530362" Nov 03 09:59:53.991437 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:53.991397 5015 generic.go:356] "Generic (PLEG): container finished" podID="92a3f6eb-9737-4c55-a115-a88d1a8199ad" containerID="15d6c8524929d7e57917a1dcefbc1cbdd57a9565db058447eedcea2ef5fb0999" exitCode=0 Nov 03 09:59:53.991863 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:53.991507 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" event={"ID":"92a3f6eb-9737-4c55-a115-a88d1a8199ad","Type":"ContainerDied","Data":"15d6c8524929d7e57917a1dcefbc1cbdd57a9565db058447eedcea2ef5fb0999"} Nov 03 09:59:55.139332 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:55.139308 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:55.270263 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:55.270183 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-openshift-ca\") pod \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " Nov 03 09:59:55.270263 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:55.270252 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a3f6eb-9737-4c55-a115-a88d1a8199ad-kube-api-access-wcvbx\" (UniqueName: \"kubernetes.io/projected/92a3f6eb-9737-4c55-a115-a88d1a8199ad-kube-api-access-wcvbx\") pod \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " Nov 03 09:59:55.270514 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:55.270373 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-xks-ca\") pod \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\" (UID: \"92a3f6eb-9737-4c55-a115-a88d1a8199ad\") " Nov 03 09:59:55.270672 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:55.270640 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-openshift-ca" pod "92a3f6eb-9737-4c55-a115-a88d1a8199ad" (UID: "92a3f6eb-9737-4c55-a115-a88d1a8199ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:59:55.270851 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:55.270826 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:55.272744 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:55.272708 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a3f6eb-9737-4c55-a115-a88d1a8199ad-kube-api-access-wcvbx" pod "92a3f6eb-9737-4c55-a115-a88d1a8199ad" (UID: "92a3f6eb-9737-4c55-a115-a88d1a8199ad"). InnerVolumeSpecName "kube-api-access-wcvbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:59:55.273088 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:55.273064 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-xks-ca" pod "92a3f6eb-9737-4c55-a115-a88d1a8199ad" (UID: "92a3f6eb-9737-4c55-a115-a88d1a8199ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:59:55.371773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:55.371727 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvbx\" (UniqueName: \"kubernetes.io/projected/92a3f6eb-9737-4c55-a115-a88d1a8199ad-kube-api-access-wcvbx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:55.371773 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:55.371759 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a3f6eb-9737-4c55-a115-a88d1a8199ad-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:56.004737 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:56.004705 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" Nov 03 09:59:56.004956 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:56.004704 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs" event={"ID":"92a3f6eb-9737-4c55-a115-a88d1a8199ad","Type":"ContainerDied","Data":"aff93d985160b5cf73a24029952a32bae3c2761884d7482b7f7018672f3784be"} Nov 03 09:59:56.004956 ip-10-0-131-14 kubenswrapper[5015]: I1103 09:59:56.004834 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aff93d985160b5cf73a24029952a32bae3c2761884d7482b7f7018672f3784be" Nov 03 10:00:00.169272 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.169240 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc"] Nov 03 10:00:00.169787 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.169696 5015 cpu_manager.go:414] "containerMap: removing container" podUID="92a3f6eb-9737-4c55-a115-a88d1a8199ad" containerName="revoke-keys" Nov 03 10:00:00.169787 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.169708 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a3f6eb-9737-4c55-a115-a88d1a8199ad" containerName="revoke-keys" Nov 03 10:00:00.169787 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.169783 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a3f6eb-9737-4c55-a115-a88d1a8199ad" containerName="revoke-keys" Nov 03 10:00:00.173132 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.173116 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" Nov 03 10:00:00.229575 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.229540 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc"] Nov 03 10:00:00.276932 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.276896 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:00:00.323495 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.323440 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16af2e82-bb6a-4077-a92c-5dc48426bd77-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895000-46dzc\" (UID: \"16af2e82-bb6a-4077-a92c-5dc48426bd77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" Nov 03 10:00:00.323671 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.323600 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq4dr\" (UniqueName: \"kubernetes.io/projected/16af2e82-bb6a-4077-a92c-5dc48426bd77-kube-api-access-lq4dr\") pod \"maas-api-key-cleanup-29895000-46dzc\" (UID: \"16af2e82-bb6a-4077-a92c-5dc48426bd77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" Nov 03 10:00:00.424293 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.424201 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16af2e82-bb6a-4077-a92c-5dc48426bd77-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895000-46dzc\" (UID: \"16af2e82-bb6a-4077-a92c-5dc48426bd77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" Nov 03 10:00:00.424293 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.424280 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq4dr\" (UniqueName: \"kubernetes.io/projected/16af2e82-bb6a-4077-a92c-5dc48426bd77-kube-api-access-lq4dr\") pod \"maas-api-key-cleanup-29895000-46dzc\" (UID: \"16af2e82-bb6a-4077-a92c-5dc48426bd77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" Nov 03 10:00:00.424876 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.424853 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16af2e82-bb6a-4077-a92c-5dc48426bd77-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895000-46dzc\" (UID: \"16af2e82-bb6a-4077-a92c-5dc48426bd77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" Nov 03 10:00:00.484078 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.484049 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq4dr\" (UniqueName: \"kubernetes.io/projected/16af2e82-bb6a-4077-a92c-5dc48426bd77-kube-api-access-lq4dr\") pod \"maas-api-key-cleanup-29895000-46dzc\" (UID: \"16af2e82-bb6a-4077-a92c-5dc48426bd77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" Nov 03 10:00:00.486860 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.486844 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" Nov 03 10:00:00.862446 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:00.862422 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc"] Nov 03 10:00:00.864841 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:00:00.864811 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16af2e82_bb6a_4077_a92c_5dc48426bd77.slice/crio-3431269ea9e41513411ffe2d02c299bf9e4d3578a1482e4af4d4f27274a6d357 WatchSource:0}: Error finding container 3431269ea9e41513411ffe2d02c299bf9e4d3578a1482e4af4d4f27274a6d357: Status 404 returned error can't find the container with id 3431269ea9e41513411ffe2d02c299bf9e4d3578a1482e4af4d4f27274a6d357 Nov 03 10:00:01.036206 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:01.036168 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" event={"ID":"16af2e82-bb6a-4077-a92c-5dc48426bd77","Type":"ContainerStarted","Data":"3431269ea9e41513411ffe2d02c299bf9e4d3578a1482e4af4d4f27274a6d357"} Nov 03 10:00:02.043723 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:02.043688 5015 generic.go:356] "Generic (PLEG): container finished" podID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerID="b756dedece03c21522d2736721ef15361bbb6440197202eb4ab1830d5eaefe38" exitCode=60 Nov 03 10:00:02.044211 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:02.043778 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" event={"ID":"16af2e82-bb6a-4077-a92c-5dc48426bd77","Type":"ContainerDied","Data":"b756dedece03c21522d2736721ef15361bbb6440197202eb4ab1830d5eaefe38"} Nov 03 10:00:02.044211 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:02.043963 5015 scope.go:122] "RemoveContainer" containerID="b756dedece03c21522d2736721ef15361bbb6440197202eb4ab1830d5eaefe38" Nov 03 10:00:03.052116 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:03.052080 5015 generic.go:356] "Generic (PLEG): container finished" podID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerID="6c52e39f4a486e66c1dbd145c94d1b3d49421c99ff35f622e6a3329322a691d0" exitCode=60 Nov 03 10:00:03.052649 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:03.052161 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" event={"ID":"16af2e82-bb6a-4077-a92c-5dc48426bd77","Type":"ContainerDied","Data":"6c52e39f4a486e66c1dbd145c94d1b3d49421c99ff35f622e6a3329322a691d0"} Nov 03 10:00:03.052649 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:03.052227 5015 scope.go:122] "RemoveContainer" containerID="b756dedece03c21522d2736721ef15361bbb6440197202eb4ab1830d5eaefe38" Nov 03 10:00:03.052649 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:03.052555 5015 scope.go:122] "RemoveContainer" containerID="6c52e39f4a486e66c1dbd145c94d1b3d49421c99ff35f622e6a3329322a691d0" Nov 03 10:00:03.052858 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:00:03.052837 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895000-46dzc_odh-ai-gateway-infra(16af2e82-bb6a-4077-a92c-5dc48426bd77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" Nov 03 10:00:04.059404 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:04.059377 5015 scope.go:122] "RemoveContainer" containerID="6c52e39f4a486e66c1dbd145c94d1b3d49421c99ff35f622e6a3329322a691d0" Nov 03 10:00:04.059898 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:00:04.059668 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895000-46dzc_odh-ai-gateway-infra(16af2e82-bb6a-4077-a92c-5dc48426bd77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" Nov 03 10:00:16.196552 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:16.196520 5015 scope.go:122] "RemoveContainer" containerID="6c52e39f4a486e66c1dbd145c94d1b3d49421c99ff35f622e6a3329322a691d0" Nov 03 10:00:16.901425 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:00:16.901389 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:00:17.135566 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:17.135532 5015 generic.go:356] "Generic (PLEG): container finished" podID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerID="14a384c2abbc13de066a81d105de5f7407ad9fe79a1a575ef3c2af330ddbc657" exitCode=60 Nov 03 10:00:17.135737 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:17.135603 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" event={"ID":"16af2e82-bb6a-4077-a92c-5dc48426bd77","Type":"ContainerDied","Data":"14a384c2abbc13de066a81d105de5f7407ad9fe79a1a575ef3c2af330ddbc657"} Nov 03 10:00:17.135737 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:17.135668 5015 scope.go:122] "RemoveContainer" containerID="6c52e39f4a486e66c1dbd145c94d1b3d49421c99ff35f622e6a3329322a691d0" Nov 03 10:00:17.136036 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:17.136012 5015 scope.go:122] "RemoveContainer" containerID="14a384c2abbc13de066a81d105de5f7407ad9fe79a1a575ef3c2af330ddbc657" Nov 03 10:00:17.136299 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:00:17.136277 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895000-46dzc_odh-ai-gateway-infra(16af2e82-bb6a-4077-a92c-5dc48426bd77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" Nov 03 10:00:18.264806 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:18.264772 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc"] Nov 03 10:00:18.407280 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:18.407249 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" Nov 03 10:00:18.595799 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:18.595723 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16af2e82-bb6a-4077-a92c-5dc48426bd77-kube-api-access-lq4dr\" (UniqueName: \"kubernetes.io/projected/16af2e82-bb6a-4077-a92c-5dc48426bd77-kube-api-access-lq4dr\") pod \"16af2e82-bb6a-4077-a92c-5dc48426bd77\" (UID: \"16af2e82-bb6a-4077-a92c-5dc48426bd77\") " Nov 03 10:00:18.595957 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:18.595865 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16af2e82-bb6a-4077-a92c-5dc48426bd77-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16af2e82-bb6a-4077-a92c-5dc48426bd77-maas-api-service-ca\") pod \"16af2e82-bb6a-4077-a92c-5dc48426bd77\" (UID: \"16af2e82-bb6a-4077-a92c-5dc48426bd77\") " Nov 03 10:00:18.596225 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:18.596199 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16af2e82-bb6a-4077-a92c-5dc48426bd77-maas-api-service-ca" pod "16af2e82-bb6a-4077-a92c-5dc48426bd77" (UID: "16af2e82-bb6a-4077-a92c-5dc48426bd77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:00:18.598175 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:18.598145 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16af2e82-bb6a-4077-a92c-5dc48426bd77-kube-api-access-lq4dr" pod "16af2e82-bb6a-4077-a92c-5dc48426bd77" (UID: "16af2e82-bb6a-4077-a92c-5dc48426bd77"). InnerVolumeSpecName "kube-api-access-lq4dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:00:18.696953 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:18.696920 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq4dr\" (UniqueName: \"kubernetes.io/projected/16af2e82-bb6a-4077-a92c-5dc48426bd77-kube-api-access-lq4dr\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:00:18.696953 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:18.696952 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16af2e82-bb6a-4077-a92c-5dc48426bd77-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:00:19.150439 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:19.150408 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" Nov 03 10:00:19.150640 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:19.150407 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc" event={"ID":"16af2e82-bb6a-4077-a92c-5dc48426bd77","Type":"ContainerDied","Data":"3431269ea9e41513411ffe2d02c299bf9e4d3578a1482e4af4d4f27274a6d357"} Nov 03 10:00:19.150640 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:19.150589 5015 scope.go:122] "RemoveContainer" containerID="14a384c2abbc13de066a81d105de5f7407ad9fe79a1a575ef3c2af330ddbc657" Nov 03 10:00:19.318680 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:19.318647 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc"] Nov 03 10:00:19.324685 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:19.324659 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895000-46dzc"] Nov 03 10:00:21.202727 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:21.202685 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" path="/var/lib/kubelet/pods/16af2e82-bb6a-4077-a92c-5dc48426bd77/volumes" Nov 03 10:00:36.778167 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.778133 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5"] Nov 03 10:00:36.778668 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.778562 5015 cpu_manager.go:414] "containerMap: removing container" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerName="cleanup" Nov 03 10:00:36.778668 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.778577 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerName="cleanup" Nov 03 10:00:36.778668 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.778606 5015 cpu_manager.go:414] "containerMap: removing container" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerName="cleanup" Nov 03 10:00:36.778668 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.778611 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerName="cleanup" Nov 03 10:00:36.778668 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.778617 5015 cpu_manager.go:414] "containerMap: removing container" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerName="cleanup" Nov 03 10:00:36.778668 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.778621 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerName="cleanup" Nov 03 10:00:36.778881 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.778700 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerName="cleanup" Nov 03 10:00:36.778881 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.778708 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerName="cleanup" Nov 03 10:00:36.778881 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.778718 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="16af2e82-bb6a-4077-a92c-5dc48426bd77" containerName="cleanup" Nov 03 10:00:36.781869 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.781853 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:36.839238 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.839197 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5"] Nov 03 10:00:36.872914 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.872875 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtdl\" (UniqueName: \"kubernetes.io/projected/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-kube-api-access-jjtdl\") pod \"maas-api-revoke-sub-f904bff9-ww8m5\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:36.873102 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.872927 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-ww8m5\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:36.873102 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.873008 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-ww8m5\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:36.885888 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.885858 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:00:36.885888 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.885859 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:00:36.974281 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.974243 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtdl\" (UniqueName: \"kubernetes.io/projected/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-kube-api-access-jjtdl\") pod \"maas-api-revoke-sub-f904bff9-ww8m5\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:36.974281 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.974290 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-ww8m5\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:36.974558 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.974311 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-ww8m5\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:36.974964 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.974944 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-ww8m5\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:36.977025 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:36.977000 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-ww8m5\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:37.034470 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:37.034389 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtdl\" (UniqueName: \"kubernetes.io/projected/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-kube-api-access-jjtdl\") pod \"maas-api-revoke-sub-f904bff9-ww8m5\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:37.095634 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:37.095594 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:37.475370 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:37.475305 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5"] Nov 03 10:00:38.263029 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:38.262986 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" event={"ID":"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21","Type":"ContainerStarted","Data":"1949e673b05e6ad443e70c2ad38bc4e0485c582e37ed67bf0b3a48217d23653d"} Nov 03 10:00:38.263415 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:38.263048 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" event={"ID":"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21","Type":"ContainerStarted","Data":"5058db1dd1254a99f8d1bfd6476b10b9f2a5be22c48f622d03882d721ffb5778"} Nov 03 10:00:38.380022 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:38.379962 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" podStartSLOduration=2.379946563 podStartE2EDuration="2.379946563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:00:38.378186198 +0000 UTC m=+4871.303128518" watchObservedRunningTime="2026-11-03 10:00:38.379946563 +0000 UTC m=+4871.304888779" Nov 03 10:00:39.270205 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:39.270166 5015 generic.go:356] "Generic (PLEG): container finished" podID="9c9236d8-ee3b-4fde-9edf-0402b6c8fe21" containerID="1949e673b05e6ad443e70c2ad38bc4e0485c582e37ed67bf0b3a48217d23653d" exitCode=0 Nov 03 10:00:39.270660 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:39.270258 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" event={"ID":"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21","Type":"ContainerDied","Data":"1949e673b05e6ad443e70c2ad38bc4e0485c582e37ed67bf0b3a48217d23653d"} Nov 03 10:00:40.416054 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:40.416029 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:40.507038 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:40.506995 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-xks-ca\") pod \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " Nov 03 10:00:40.507239 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:40.507056 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-kube-api-access-jjtdl\" (UniqueName: \"kubernetes.io/projected/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-kube-api-access-jjtdl\") pod \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " Nov 03 10:00:40.507239 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:40.507083 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-openshift-ca\") pod \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\" (UID: \"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21\") " Nov 03 10:00:40.507524 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:40.507464 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-openshift-ca" pod "9c9236d8-ee3b-4fde-9edf-0402b6c8fe21" (UID: "9c9236d8-ee3b-4fde-9edf-0402b6c8fe21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:00:40.509466 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:40.509438 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-kube-api-access-jjtdl" pod "9c9236d8-ee3b-4fde-9edf-0402b6c8fe21" (UID: "9c9236d8-ee3b-4fde-9edf-0402b6c8fe21"). InnerVolumeSpecName "kube-api-access-jjtdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:00:40.509466 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:40.509455 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-xks-ca" pod "9c9236d8-ee3b-4fde-9edf-0402b6c8fe21" (UID: "9c9236d8-ee3b-4fde-9edf-0402b6c8fe21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:00:40.608249 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:40.608167 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:00:40.608249 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:40.608196 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjtdl\" (UniqueName: \"kubernetes.io/projected/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-kube-api-access-jjtdl\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:00:40.608249 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:40.608207 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:00:41.283586 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:41.283542 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" event={"ID":"9c9236d8-ee3b-4fde-9edf-0402b6c8fe21","Type":"ContainerDied","Data":"5058db1dd1254a99f8d1bfd6476b10b9f2a5be22c48f622d03882d721ffb5778"} Nov 03 10:00:41.283586 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:41.283568 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5" Nov 03 10:00:41.283586 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:00:41.283595 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5058db1dd1254a99f8d1bfd6476b10b9f2a5be22c48f622d03882d721ffb5778" Nov 03 10:00:46.902606 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:00:46.902561 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:01:02.289904 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.289867 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm"] Nov 03 10:01:02.290346 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.290331 5015 cpu_manager.go:414] "containerMap: removing container" podUID="9c9236d8-ee3b-4fde-9edf-0402b6c8fe21" containerName="revoke-keys" Nov 03 10:01:02.290396 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.290348 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9236d8-ee3b-4fde-9edf-0402b6c8fe21" containerName="revoke-keys" Nov 03 10:01:02.290433 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.290424 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9236d8-ee3b-4fde-9edf-0402b6c8fe21" containerName="revoke-keys" Nov 03 10:01:02.293730 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.293709 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.340145 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.340082 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc"] Nov 03 10:01:02.344017 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.343995 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.392621 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.392587 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm"] Nov 03 10:01:02.403213 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.403180 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc"] Nov 03 10:01:02.421270 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.421236 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.421433 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.421365 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.421433 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.421396 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.421605 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.421493 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbwx5\" (UniqueName: \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-kube-api-access-bbwx5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.421605 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.421529 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.421711 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.421636 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.421711 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.421683 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.421792 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.421749 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.421792 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.421779 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523019 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.522982 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523019 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523021 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523278 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523047 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.523278 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523082 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded8f558-e327-4178-a2ac-948e7f005ef1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.523278 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523113 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.523278 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523137 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523278 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523163 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.523511 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523354 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523511 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523385 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523511 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523416 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523511 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523434 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523661 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523514 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.523661 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523562 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523661 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523586 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523661 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523612 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523661 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523634 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.523924 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523669 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbwx5\" (UniqueName: \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-kube-api-access-bbwx5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523924 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523732 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf5m8\" (UniqueName: \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-kube-api-access-hf5m8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.523924 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523761 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523924 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523772 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.523924 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523802 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.524249 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.523914 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.524249 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.524203 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.525941 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.525910 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.526363 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.526345 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.583769 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.583686 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.583769 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.583748 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbwx5\" (UniqueName: \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-kube-api-access-bbwx5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lg5gm\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.607060 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.607033 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:02.624524 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.624497 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.624672 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.624551 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.624672 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.624582 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded8f558-e327-4178-a2ac-948e7f005ef1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.624672 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.624605 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.624672 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.624627 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.624918 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.624690 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.624918 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.624738 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.624918 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.624783 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf5m8\" (UniqueName: \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-kube-api-access-hf5m8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.624918 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.624814 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.625174 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.624919 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.625671 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.625646 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.625837 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.625701 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded8f558-e327-4178-a2ac-948e7f005ef1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.625930 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.625906 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.626233 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.626215 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.628299 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.628271 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.628538 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.628521 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.685789 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.685760 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.686657 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.686634 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf5m8\" (UniqueName: \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-kube-api-access-hf5m8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-dhjmc\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:02.958673 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:02.958596 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:03.042824 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.042793 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm"] Nov 03 10:01:03.043626 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:01:03.043592 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1bcdca_f1fe_45db_98df_66f3ebd5696e.slice/crio-fc9e4f4452b8e941ee13b38f7bc97b64512800eaab2e3fbe4c9362854b4729ed WatchSource:0}: Error finding container fc9e4f4452b8e941ee13b38f7bc97b64512800eaab2e3fbe4c9362854b4729ed: Status 404 returned error can't find the container with id fc9e4f4452b8e941ee13b38f7bc97b64512800eaab2e3fbe4c9362854b4729ed Nov 03 10:01:03.046637 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.046598 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:01:03.046740 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.046690 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:01:03.046740 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.046733 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:01:03.338578 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.338547 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc"] Nov 03 10:01:03.339755 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:01:03.339715 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded8f558_e327_4178_a2ac_948e7f005ef1.slice/crio-61fa884327a0f4d80d50c74e32fbf76a81651ca08bcc00c4c9a73219eb2bbb3d WatchSource:0}: Error finding container 61fa884327a0f4d80d50c74e32fbf76a81651ca08bcc00c4c9a73219eb2bbb3d: Status 404 returned error can't find the container with id 61fa884327a0f4d80d50c74e32fbf76a81651ca08bcc00c4c9a73219eb2bbb3d Nov 03 10:01:03.342779 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.342740 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:01:03.342888 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.342831 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:01:03.342888 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.342879 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:01:03.416542 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.416512 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" event={"ID":"0d1bcdca-f1fe-45db-98df-66f3ebd5696e","Type":"ContainerStarted","Data":"0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd"} Nov 03 10:01:03.416682 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.416565 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" event={"ID":"0d1bcdca-f1fe-45db-98df-66f3ebd5696e","Type":"ContainerStarted","Data":"fc9e4f4452b8e941ee13b38f7bc97b64512800eaab2e3fbe4c9362854b4729ed"} Nov 03 10:01:03.418454 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.418422 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" event={"ID":"ded8f558-e327-4178-a2ac-948e7f005ef1","Type":"ContainerStarted","Data":"61fa884327a0f4d80d50c74e32fbf76a81651ca08bcc00c4c9a73219eb2bbb3d"} Nov 03 10:01:03.537097 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.537028 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podStartSLOduration=1.537010089 podStartE2EDuration="1.537010089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:01:03.535920833 +0000 UTC m=+4896.460863049" watchObservedRunningTime="2026-11-03 10:01:03.537010089 +0000 UTC m=+4896.461952307" Nov 03 10:01:03.607411 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.607313 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:03.668931 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.668860 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podStartSLOduration=1.6688376150000002 podStartE2EDuration="1.668837615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:01:03.654222147 +0000 UTC m=+4896.579164377" watchObservedRunningTime="2026-11-03 10:01:03.668837615 +0000 UTC m=+4896.593779833" Nov 03 10:01:03.959557 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:03.959057 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:04.425628 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:04.425587 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" event={"ID":"ded8f558-e327-4178-a2ac-948e7f005ef1","Type":"ContainerStarted","Data":"edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690"} Nov 03 10:01:04.608897 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:04.608858 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Nov 03 10:01:04.609075 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:04.608932 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Nov 03 10:01:04.966502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:04.966063 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 10:01:04.966502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:04.966142 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 10:01:05.610503 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:05.608946 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Nov 03 10:01:05.610503 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:05.609031 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Nov 03 10:01:05.993511 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:05.973621 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 10:01:05.993511 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:05.973715 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 10:01:06.608536 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:06.608119 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Nov 03 10:01:06.608536 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:06.608228 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Nov 03 10:01:06.968556 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:06.968405 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 10:01:06.968556 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:06.968467 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 10:01:07.608880 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:07.608847 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Nov 03 10:01:07.609109 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:07.608907 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Nov 03 10:01:07.961979 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:07.961886 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 10:01:07.962361 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:07.962332 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 10:01:08.609987 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:08.609946 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Nov 03 10:01:08.610830 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:08.610795 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Nov 03 10:01:08.961570 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:08.961013 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 10:01:08.961570 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:08.961092 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 10:01:09.608393 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:09.608337 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Nov 03 10:01:09.608664 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:09.608426 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Nov 03 10:01:09.960040 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:09.959951 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 10:01:09.960443 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:09.960035 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 10:01:10.608341 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:10.608297 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Nov 03 10:01:10.608580 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:10.608368 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Nov 03 10:01:10.959694 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:10.959594 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 10:01:10.959694 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:10.959673 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 10:01:10.986382 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:10.986350 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4"] Nov 03 10:01:10.990346 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:10.990325 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:11.050901 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.050863 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4"] Nov 03 10:01:11.066721 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.066689 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:01:11.096674 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.096637 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:01:11.125109 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.125073 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7xsc4\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:11.125109 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.125118 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7xsc4\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:11.125408 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.125320 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxphh\" (UniqueName: \"kubernetes.io/projected/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-kube-api-access-zxphh\") pod \"maas-api-revoke-sub-f03629db-7xsc4\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:11.226233 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.226200 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxphh\" (UniqueName: \"kubernetes.io/projected/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-kube-api-access-zxphh\") pod \"maas-api-revoke-sub-f03629db-7xsc4\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:11.226442 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.226311 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7xsc4\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:11.226442 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.226342 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7xsc4\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:11.227321 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.227290 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7xsc4\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:11.228970 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.228945 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7xsc4\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:11.286294 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.286258 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxphh\" (UniqueName: \"kubernetes.io/projected/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-kube-api-access-zxphh\") pod \"maas-api-revoke-sub-f03629db-7xsc4\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:11.310859 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.310812 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:11.702765 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:11.702215 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4"] Nov 03 10:01:12.527540 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:12.527497 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" event={"ID":"f10edf53-264a-4cdf-82e1-6454ec1c7fb0","Type":"ContainerStarted","Data":"d767d0298938972c4f9e53a90b319d963c5470d63847ba2c2bd036e65e85863e"} Nov 03 10:01:12.527990 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:12.527572 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" event={"ID":"f10edf53-264a-4cdf-82e1-6454ec1c7fb0","Type":"ContainerStarted","Data":"d137ad0738323610f8b770f7cf5ea28bdc96dcc5c4fff8bbc99291f7c671e1fc"} Nov 03 10:01:12.607804 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:12.607757 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:01:12.608012 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:12.607825 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:01:12.961022 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:12.960846 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:01:12.961022 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:12.960908 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:01:13.534984 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:13.534943 5015 generic.go:356] "Generic (PLEG): container finished" podID="f10edf53-264a-4cdf-82e1-6454ec1c7fb0" containerID="d767d0298938972c4f9e53a90b319d963c5470d63847ba2c2bd036e65e85863e" exitCode=0 Nov 03 10:01:13.535394 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:13.534995 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" event={"ID":"f10edf53-264a-4cdf-82e1-6454ec1c7fb0","Type":"ContainerDied","Data":"d767d0298938972c4f9e53a90b319d963c5470d63847ba2c2bd036e65e85863e"} Nov 03 10:01:13.656442 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:13.656391 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:01:13.656797 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:13.656747 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:01:14.055722 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.055644 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:01:14.055940 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.055753 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:01:14.458249 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.457672 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:14.458534 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.458512 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:14.459346 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.459307 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:01:14.705011 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.704983 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:14.723192 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.723171 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-xks-ca\") pod \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " Nov 03 10:01:14.723308 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.723214 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-openshift-ca\") pod \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " Nov 03 10:01:14.723352 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.723335 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-kube-api-access-zxphh\" (UniqueName: \"kubernetes.io/projected/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-kube-api-access-zxphh\") pod \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\" (UID: \"f10edf53-264a-4cdf-82e1-6454ec1c7fb0\") " Nov 03 10:01:14.723720 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.723694 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-openshift-ca" pod "f10edf53-264a-4cdf-82e1-6454ec1c7fb0" (UID: "f10edf53-264a-4cdf-82e1-6454ec1c7fb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:01:14.725507 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.725445 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-xks-ca" pod "f10edf53-264a-4cdf-82e1-6454ec1c7fb0" (UID: "f10edf53-264a-4cdf-82e1-6454ec1c7fb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:01:14.725687 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.725663 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-kube-api-access-zxphh" pod "f10edf53-264a-4cdf-82e1-6454ec1c7fb0" (UID: "f10edf53-264a-4cdf-82e1-6454ec1c7fb0"). InnerVolumeSpecName "kube-api-access-zxphh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:01:14.777902 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.777868 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:14.778295 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.778264 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:14.778955 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.778935 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:01:14.824366 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.824281 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:01:14.824366 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.824317 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxphh\" (UniqueName: \"kubernetes.io/projected/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-kube-api-access-zxphh\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:01:14.824366 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:14.824335 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10edf53-264a-4cdf-82e1-6454ec1c7fb0-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:01:15.550950 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:15.550918 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" Nov 03 10:01:15.551138 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:15.550958 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4" event={"ID":"f10edf53-264a-4cdf-82e1-6454ec1c7fb0","Type":"ContainerDied","Data":"d137ad0738323610f8b770f7cf5ea28bdc96dcc5c4fff8bbc99291f7c671e1fc"} Nov 03 10:01:15.551138 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:15.551015 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d137ad0738323610f8b770f7cf5ea28bdc96dcc5c4fff8bbc99291f7c671e1fc" Nov 03 10:01:16.902341 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:01:16.902295 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:01:30.515615 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:30.515572 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:01:30.516258 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:30.516228 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:01:30.826750 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:30.826639 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:01:30.826750 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:30.826726 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:01:31.841573 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:31.841538 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595"] Nov 03 10:01:31.843942 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:31.841975 5015 cpu_manager.go:414] "containerMap: removing container" podUID="f10edf53-264a-4cdf-82e1-6454ec1c7fb0" containerName="revoke-keys" Nov 03 10:01:31.843942 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:31.841986 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10edf53-264a-4cdf-82e1-6454ec1c7fb0" containerName="revoke-keys" Nov 03 10:01:31.843942 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:31.842048 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10edf53-264a-4cdf-82e1-6454ec1c7fb0" containerName="revoke-keys" Nov 03 10:01:31.845060 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:31.845035 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:31.903441 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:31.903404 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595"] Nov 03 10:01:31.941847 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:31.941811 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dp595\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:31.941847 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:31.941846 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dp595\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:31.942109 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:31.941991 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59pkj\" (UniqueName: \"kubernetes.io/projected/50a66ec3-51b9-425e-82e9-210cce020726-kube-api-access-59pkj\") pod \"maas-api-revoke-sub-0165d7d2-dp595\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:31.949175 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:31.949153 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:01:31.949298 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:31.949226 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:01:32.043062 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:32.043029 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59pkj\" (UniqueName: \"kubernetes.io/projected/50a66ec3-51b9-425e-82e9-210cce020726-kube-api-access-59pkj\") pod \"maas-api-revoke-sub-0165d7d2-dp595\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:32.043245 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:32.043131 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dp595\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:32.043245 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:32.043162 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dp595\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:32.043943 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:32.043925 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dp595\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:32.045732 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:32.045712 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dp595\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:32.103049 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:32.102980 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59pkj\" (UniqueName: \"kubernetes.io/projected/50a66ec3-51b9-425e-82e9-210cce020726-kube-api-access-59pkj\") pod \"maas-api-revoke-sub-0165d7d2-dp595\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:32.158557 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:32.158528 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:32.534368 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:32.534337 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595"] Nov 03 10:01:32.535214 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:01:32.535186 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a66ec3_51b9_425e_82e9_210cce020726.slice/crio-5af6fbb4233b57726be00f8f48ca3b9223269ae5b7b2d8a463812fd6a54beee9 WatchSource:0}: Error finding container 5af6fbb4233b57726be00f8f48ca3b9223269ae5b7b2d8a463812fd6a54beee9: Status 404 returned error can't find the container with id 5af6fbb4233b57726be00f8f48ca3b9223269ae5b7b2d8a463812fd6a54beee9 Nov 03 10:01:32.728440 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:32.728405 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" event={"ID":"50a66ec3-51b9-425e-82e9-210cce020726","Type":"ContainerStarted","Data":"5af6fbb4233b57726be00f8f48ca3b9223269ae5b7b2d8a463812fd6a54beee9"} Nov 03 10:01:33.735782 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:33.735748 5015 generic.go:356] "Generic (PLEG): container finished" podID="50a66ec3-51b9-425e-82e9-210cce020726" containerID="622e2bc20d285c4a0cda126b86c2f136b44f2adf2dddefb197de12788c66fc41" exitCode=0 Nov 03 10:01:33.736200 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:33.735831 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" event={"ID":"50a66ec3-51b9-425e-82e9-210cce020726","Type":"ContainerDied","Data":"622e2bc20d285c4a0cda126b86c2f136b44f2adf2dddefb197de12788c66fc41"} Nov 03 10:01:34.877760 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:34.877733 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:34.974116 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:34.974080 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a66ec3-51b9-425e-82e9-210cce020726-kube-api-access-59pkj\" (UniqueName: \"kubernetes.io/projected/50a66ec3-51b9-425e-82e9-210cce020726-kube-api-access-59pkj\") pod \"50a66ec3-51b9-425e-82e9-210cce020726\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " Nov 03 10:01:34.974291 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:34.974143 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-openshift-ca\") pod \"50a66ec3-51b9-425e-82e9-210cce020726\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " Nov 03 10:01:34.974291 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:34.974179 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-xks-ca\") pod \"50a66ec3-51b9-425e-82e9-210cce020726\" (UID: \"50a66ec3-51b9-425e-82e9-210cce020726\") " Nov 03 10:01:34.974586 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:34.974559 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-openshift-ca" pod "50a66ec3-51b9-425e-82e9-210cce020726" (UID: "50a66ec3-51b9-425e-82e9-210cce020726"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:01:34.976567 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:34.976536 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-xks-ca" pod "50a66ec3-51b9-425e-82e9-210cce020726" (UID: "50a66ec3-51b9-425e-82e9-210cce020726"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:01:34.976692 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:34.976612 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a66ec3-51b9-425e-82e9-210cce020726-kube-api-access-59pkj" pod "50a66ec3-51b9-425e-82e9-210cce020726" (UID: "50a66ec3-51b9-425e-82e9-210cce020726"). InnerVolumeSpecName "kube-api-access-59pkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:01:35.075147 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:35.075064 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59pkj\" (UniqueName: \"kubernetes.io/projected/50a66ec3-51b9-425e-82e9-210cce020726-kube-api-access-59pkj\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:01:35.075147 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:35.075095 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:01:35.075147 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:35.075107 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a66ec3-51b9-425e-82e9-210cce020726-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:01:35.749765 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:35.749731 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" Nov 03 10:01:35.749943 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:35.749767 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595" event={"ID":"50a66ec3-51b9-425e-82e9-210cce020726","Type":"ContainerDied","Data":"5af6fbb4233b57726be00f8f48ca3b9223269ae5b7b2d8a463812fd6a54beee9"} Nov 03 10:01:35.749943 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:35.749850 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5af6fbb4233b57726be00f8f48ca3b9223269ae5b7b2d8a463812fd6a54beee9" Nov 03 10:01:43.672952 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.672915 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr"] Nov 03 10:01:43.673353 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.673321 5015 cpu_manager.go:414] "containerMap: removing container" podUID="50a66ec3-51b9-425e-82e9-210cce020726" containerName="revoke-keys" Nov 03 10:01:43.673353 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.673332 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a66ec3-51b9-425e-82e9-210cce020726" containerName="revoke-keys" Nov 03 10:01:43.673432 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.673419 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a66ec3-51b9-425e-82e9-210cce020726" containerName="revoke-keys" Nov 03 10:01:43.676685 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.676667 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:43.734252 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.734220 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr"] Nov 03 10:01:43.765177 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.765145 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mdbpr\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:43.765360 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.765181 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbs28\" (UniqueName: \"kubernetes.io/projected/f3e27578-79ff-4c2e-868d-f954d653c665-kube-api-access-vbs28\") pod \"maas-api-revoke-sub-6b19338f-mdbpr\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:43.765360 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.765212 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mdbpr\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:43.780820 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.780791 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:01:43.780962 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.780829 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:01:43.866444 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.866400 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbs28\" (UniqueName: \"kubernetes.io/projected/f3e27578-79ff-4c2e-868d-f954d653c665-kube-api-access-vbs28\") pod \"maas-api-revoke-sub-6b19338f-mdbpr\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:43.866444 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.866442 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mdbpr\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:43.866736 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.866559 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mdbpr\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:43.867291 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.867256 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mdbpr\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:43.869153 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.869128 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mdbpr\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:43.926021 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.925944 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbs28\" (UniqueName: \"kubernetes.io/projected/f3e27578-79ff-4c2e-868d-f954d653c665-kube-api-access-vbs28\") pod \"maas-api-revoke-sub-6b19338f-mdbpr\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:43.990057 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:43.990021 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:44.374907 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:01:44.374867 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e27578_79ff_4c2e_868d_f954d653c665.slice/crio-259a98c73b726bc532669bbc641805a247a2d6e576afbba696d32a2cb002d2d2 WatchSource:0}: Error finding container 259a98c73b726bc532669bbc641805a247a2d6e576afbba696d32a2cb002d2d2: Status 404 returned error can't find the container with id 259a98c73b726bc532669bbc641805a247a2d6e576afbba696d32a2cb002d2d2 Nov 03 10:01:44.375054 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:44.374916 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr"] Nov 03 10:01:44.813938 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:44.813905 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" event={"ID":"f3e27578-79ff-4c2e-868d-f954d653c665","Type":"ContainerStarted","Data":"1ff0b6502c251e73c836336ed609fa2a32f56b8b225b0edef6f96f1a24559762"} Nov 03 10:01:44.814428 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:44.813963 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" event={"ID":"f3e27578-79ff-4c2e-868d-f954d653c665","Type":"ContainerStarted","Data":"259a98c73b726bc532669bbc641805a247a2d6e576afbba696d32a2cb002d2d2"} Nov 03 10:01:44.931001 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:44.930872 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" podStartSLOduration=1.9308568990000001 podStartE2EDuration="1.930856899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:01:44.929464574 +0000 UTC m=+4937.854406804" watchObservedRunningTime="2026-11-03 10:01:44.930856899 +0000 UTC m=+4937.855799115" Nov 03 10:01:45.821095 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:45.821063 5015 generic.go:356] "Generic (PLEG): container finished" podID="f3e27578-79ff-4c2e-868d-f954d653c665" containerID="1ff0b6502c251e73c836336ed609fa2a32f56b8b225b0edef6f96f1a24559762" exitCode=0 Nov 03 10:01:45.821542 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:45.821151 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" event={"ID":"f3e27578-79ff-4c2e-868d-f954d653c665","Type":"ContainerDied","Data":"1ff0b6502c251e73c836336ed609fa2a32f56b8b225b0edef6f96f1a24559762"} Nov 03 10:01:46.907082 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:01:46.907044 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:01:46.966027 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:46.966002 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:47.099698 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.099613 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-openshift-ca\") pod \"f3e27578-79ff-4c2e-868d-f954d653c665\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " Nov 03 10:01:47.099698 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.099671 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-xks-ca\") pod \"f3e27578-79ff-4c2e-868d-f954d653c665\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " Nov 03 10:01:47.099896 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.099758 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e27578-79ff-4c2e-868d-f954d653c665-kube-api-access-vbs28\" (UniqueName: \"kubernetes.io/projected/f3e27578-79ff-4c2e-868d-f954d653c665-kube-api-access-vbs28\") pod \"f3e27578-79ff-4c2e-868d-f954d653c665\" (UID: \"f3e27578-79ff-4c2e-868d-f954d653c665\") " Nov 03 10:01:47.100000 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.099978 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-openshift-ca" pod "f3e27578-79ff-4c2e-868d-f954d653c665" (UID: "f3e27578-79ff-4c2e-868d-f954d653c665"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:01:47.100224 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.100200 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:01:47.102072 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.102047 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-xks-ca" pod "f3e27578-79ff-4c2e-868d-f954d653c665" (UID: "f3e27578-79ff-4c2e-868d-f954d653c665"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:01:47.102175 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.102085 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e27578-79ff-4c2e-868d-f954d653c665-kube-api-access-vbs28" pod "f3e27578-79ff-4c2e-868d-f954d653c665" (UID: "f3e27578-79ff-4c2e-868d-f954d653c665"). InnerVolumeSpecName "kube-api-access-vbs28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:01:47.203259 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.203222 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e27578-79ff-4c2e-868d-f954d653c665-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:01:47.203550 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.203531 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbs28\" (UniqueName: \"kubernetes.io/projected/f3e27578-79ff-4c2e-868d-f954d653c665-kube-api-access-vbs28\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:01:47.834876 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.834836 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" event={"ID":"f3e27578-79ff-4c2e-868d-f954d653c665","Type":"ContainerDied","Data":"259a98c73b726bc532669bbc641805a247a2d6e576afbba696d32a2cb002d2d2"} Nov 03 10:01:47.834876 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.834869 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr" Nov 03 10:01:47.835105 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:47.834889 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="259a98c73b726bc532669bbc641805a247a2d6e576afbba696d32a2cb002d2d2" Nov 03 10:01:52.058057 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:52.058020 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb"] Nov 03 10:01:52.108521 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:52.108470 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b61c7df-pbzkb"] Nov 03 10:01:53.203355 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:53.203324 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a4c0d0-ad33-41df-ac33-99f23537a233" path="/var/lib/kubelet/pods/55a4c0d0-ad33-41df-ac33-99f23537a233/volumes" Nov 03 10:01:58.280503 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.280077 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf"] Nov 03 10:01:58.281059 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.280745 5015 cpu_manager.go:414] "containerMap: removing container" podUID="f3e27578-79ff-4c2e-868d-f954d653c665" containerName="revoke-keys" Nov 03 10:01:58.281059 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.280766 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e27578-79ff-4c2e-868d-f954d653c665" containerName="revoke-keys" Nov 03 10:01:58.281059 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.280933 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e27578-79ff-4c2e-868d-f954d653c665" containerName="revoke-keys" Nov 03 10:01:58.285330 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.285305 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.324836 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.324795 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf"] Nov 03 10:01:58.392527 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.384814 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-8906966c-kserve-self-signed-certs\"" Nov 03 10:01:58.428503 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.420897 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb8lg\" (UniqueName: \"kubernetes.io/projected/15f6894d-cb7a-45c9-8490-16b8d30c08e1-kube-api-access-pb8lg\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.428503 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.421006 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-home\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.428503 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.421104 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-model-cache\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.428503 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.421147 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tls-certs\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.428503 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.421328 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-dshm\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.428503 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.421399 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tmp-dir\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.522730 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.522684 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-model-cache\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.522730 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.522727 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tls-certs\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.523065 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.522814 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-dshm\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.523065 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.522857 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tmp-dir\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.523065 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.522888 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb8lg\" (UniqueName: \"kubernetes.io/projected/15f6894d-cb7a-45c9-8490-16b8d30c08e1-kube-api-access-pb8lg\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.523065 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.522930 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-home\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.523290 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.523156 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-model-cache\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.523290 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.523252 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-home\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.523513 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.523442 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tmp-dir\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.525833 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.525809 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-dshm\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.526254 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.526231 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tls-certs\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.584122 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.584012 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb8lg\" (UniqueName: \"kubernetes.io/projected/15f6894d-cb7a-45c9-8490-16b8d30c08e1-kube-api-access-pb8lg\") pod \"e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:58.619349 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:58.619316 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:01:59.084184 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:59.075801 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf"] Nov 03 10:01:59.935952 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:59.935911 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" event={"ID":"15f6894d-cb7a-45c9-8490-16b8d30c08e1","Type":"ContainerStarted","Data":"3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859"} Nov 03 10:01:59.936457 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:01:59.935962 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" event={"ID":"15f6894d-cb7a-45c9-8490-16b8d30c08e1","Type":"ContainerStarted","Data":"69dd734d717f86301c58c03fbf59ed40b960ddb9170d3d04f00af07e555a65de"} Nov 03 10:02:00.053456 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:02:00.053390 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" podStartSLOduration=2.053375047 podStartE2EDuration="2.053375047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:02:00.052353251 +0000 UTC m=+4952.977295469" watchObservedRunningTime="2026-11-03 10:02:00.053375047 +0000 UTC m=+4952.978317264" Nov 03 10:02:00.619417 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:02:00.619380 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:02:08.620147 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:02:08.620096 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:02:08.633178 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:02:08.633149 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:02:09.040659 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:02:09.040630 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:02:12.052230 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:02:12.052196 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656"] Nov 03 10:02:12.102876 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:02:12.102845 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37c102d-6b656"] Nov 03 10:02:13.205044 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:02:13.205013 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc96401-10e3-4e01-a057-23d5c9516e1b" path="/var/lib/kubelet/pods/4bc96401-10e3-4e01-a057-23d5c9516e1b/volumes" Nov 03 10:02:16.903026 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:02:16.902990 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:02:26.052936 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:02:26.052898 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch"] Nov 03 10:02:26.102946 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:02:26.102910 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db91d8bb-4btch"] Nov 03 10:02:27.203787 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:02:27.203746 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd" path="/var/lib/kubelet/pods/0082f1b9-4cf4-4ee0-9abe-a19fa84bb5bd/volumes" Nov 03 10:02:46.902239 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:02:46.902159 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:03:13.760560 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:13.760526 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p"] Nov 03 10:03:13.764149 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:13.764131 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:13.822238 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:13.822210 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p"] Nov 03 10:03:13.868421 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:13.868390 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:03:13.868585 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:13.868491 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:03:13.946537 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:13.946509 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5549abbc-rsx9p\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:13.946724 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:13.946546 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jssx9\" (UniqueName: \"kubernetes.io/projected/82dfe9be-d230-490b-8805-e0fc32069592-kube-api-access-jssx9\") pod \"maas-api-revoke-sub-5549abbc-rsx9p\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:13.946724 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:13.946567 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5549abbc-rsx9p\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:14.048071 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:14.047972 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5549abbc-rsx9p\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:14.048071 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:14.048014 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jssx9\" (UniqueName: \"kubernetes.io/projected/82dfe9be-d230-490b-8805-e0fc32069592-kube-api-access-jssx9\") pod \"maas-api-revoke-sub-5549abbc-rsx9p\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:14.048071 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:14.048041 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5549abbc-rsx9p\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:14.048787 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:14.048761 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5549abbc-rsx9p\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:14.050589 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:14.050570 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5549abbc-rsx9p\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:14.107127 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:14.107093 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jssx9\" (UniqueName: \"kubernetes.io/projected/82dfe9be-d230-490b-8805-e0fc32069592-kube-api-access-jssx9\") pod \"maas-api-revoke-sub-5549abbc-rsx9p\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:14.377161 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:14.377071 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:14.752405 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:14.752376 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p"] Nov 03 10:03:14.753974 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:03:14.753945 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82dfe9be_d230_490b_8805_e0fc32069592.slice/crio-1002b9c04f1c9c8e25209555469178745d5bf279fa462dba21e472f64a8a2746 WatchSource:0}: Error finding container 1002b9c04f1c9c8e25209555469178745d5bf279fa462dba21e472f64a8a2746: Status 404 returned error can't find the container with id 1002b9c04f1c9c8e25209555469178745d5bf279fa462dba21e472f64a8a2746 Nov 03 10:03:15.456656 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:15.456567 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" event={"ID":"82dfe9be-d230-490b-8805-e0fc32069592","Type":"ContainerStarted","Data":"8fcffbfa3ac194766e1124968e31b3690deba3131acc4106a7d2de7dc68caee8"} Nov 03 10:03:15.456656 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:15.456630 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" event={"ID":"82dfe9be-d230-490b-8805-e0fc32069592","Type":"ContainerStarted","Data":"1002b9c04f1c9c8e25209555469178745d5bf279fa462dba21e472f64a8a2746"} Nov 03 10:03:15.572491 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:15.572425 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" podStartSLOduration=2.572411526 podStartE2EDuration="2.572411526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:03:15.57158667 +0000 UTC m=+5028.496528902" watchObservedRunningTime="2026-11-03 10:03:15.572411526 +0000 UTC m=+5028.497353743" Nov 03 10:03:16.469381 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:16.469347 5015 generic.go:356] "Generic (PLEG): container finished" podID="82dfe9be-d230-490b-8805-e0fc32069592" containerID="8fcffbfa3ac194766e1124968e31b3690deba3131acc4106a7d2de7dc68caee8" exitCode=0 Nov 03 10:03:16.469843 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:16.469537 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" event={"ID":"82dfe9be-d230-490b-8805-e0fc32069592","Type":"ContainerDied","Data":"8fcffbfa3ac194766e1124968e31b3690deba3131acc4106a7d2de7dc68caee8"} Nov 03 10:03:16.901512 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:03:16.901457 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:03:17.619468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:17.619438 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:17.680569 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:17.680541 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-openshift-ca\") pod \"82dfe9be-d230-490b-8805-e0fc32069592\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " Nov 03 10:03:17.680756 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:17.680627 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82dfe9be-d230-490b-8805-e0fc32069592-kube-api-access-jssx9\" (UniqueName: \"kubernetes.io/projected/82dfe9be-d230-490b-8805-e0fc32069592-kube-api-access-jssx9\") pod \"82dfe9be-d230-490b-8805-e0fc32069592\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " Nov 03 10:03:17.680931 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:17.680908 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-openshift-ca" pod "82dfe9be-d230-490b-8805-e0fc32069592" (UID: "82dfe9be-d230-490b-8805-e0fc32069592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:03:17.682922 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:17.682901 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dfe9be-d230-490b-8805-e0fc32069592-kube-api-access-jssx9" pod "82dfe9be-d230-490b-8805-e0fc32069592" (UID: "82dfe9be-d230-490b-8805-e0fc32069592"). InnerVolumeSpecName "kube-api-access-jssx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:03:17.781585 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:17.781462 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-xks-ca\") pod \"82dfe9be-d230-490b-8805-e0fc32069592\" (UID: \"82dfe9be-d230-490b-8805-e0fc32069592\") " Nov 03 10:03:17.781809 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:17.781747 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jssx9\" (UniqueName: \"kubernetes.io/projected/82dfe9be-d230-490b-8805-e0fc32069592-kube-api-access-jssx9\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:17.781809 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:17.781761 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:17.783896 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:17.783874 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-xks-ca" pod "82dfe9be-d230-490b-8805-e0fc32069592" (UID: "82dfe9be-d230-490b-8805-e0fc32069592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:03:17.882263 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:17.882219 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82dfe9be-d230-490b-8805-e0fc32069592-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:18.483178 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:18.483147 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" Nov 03 10:03:18.483371 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:18.483187 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p" event={"ID":"82dfe9be-d230-490b-8805-e0fc32069592","Type":"ContainerDied","Data":"1002b9c04f1c9c8e25209555469178745d5bf279fa462dba21e472f64a8a2746"} Nov 03 10:03:18.483371 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:18.483235 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1002b9c04f1c9c8e25209555469178745d5bf279fa462dba21e472f64a8a2746" Nov 03 10:03:26.763113 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:26.763079 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf"] Nov 03 10:03:29.506010 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:29.505972 5015 scope.go:122] "RemoveContainer" containerID="ff7a00fab6e6a045b3ef322d08c9f4f00b54c09c80c4ea7314347aacd6b8f7b7" Nov 03 10:03:29.515602 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:29.515576 5015 scope.go:122] "RemoveContainer" containerID="08a608d2810f326d55d467eb3d06defbf8d024857cb68b40df767578c06f5045" Nov 03 10:03:29.524596 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:29.524572 5015 scope.go:122] "RemoveContainer" containerID="d185b714626c2a8c7e2e4a5011af6c0da0d7e1ffa518c7e71651a9b7c741debc" Nov 03 10:03:32.289304 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.289264 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f"] Nov 03 10:03:32.289938 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.289909 5015 cpu_manager.go:414] "containerMap: removing container" podUID="82dfe9be-d230-490b-8805-e0fc32069592" containerName="revoke-keys" Nov 03 10:03:32.289938 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.289937 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82dfe9be-d230-490b-8805-e0fc32069592" containerName="revoke-keys" Nov 03 10:03:32.290157 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.290075 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="82dfe9be-d230-490b-8805-e0fc32069592" containerName="revoke-keys" Nov 03 10:03:32.295441 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.295418 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:32.352646 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.352609 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f"] Nov 03 10:03:32.395749 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.395716 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:03:32.395937 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.395804 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:03:32.423830 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.423793 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae69f747-qh78f\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:32.424052 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.423965 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae69f747-qh78f\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:32.424052 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.424033 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfc4l\" (UniqueName: \"kubernetes.io/projected/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-kube-api-access-kfc4l\") pod \"maas-api-revoke-sub-ae69f747-qh78f\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:32.525213 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.525173 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae69f747-qh78f\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:32.525433 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.525275 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae69f747-qh78f\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:32.525433 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.525318 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfc4l\" (UniqueName: \"kubernetes.io/projected/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-kube-api-access-kfc4l\") pod \"maas-api-revoke-sub-ae69f747-qh78f\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:32.526076 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.526052 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae69f747-qh78f\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:32.528164 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.528135 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae69f747-qh78f\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:32.588236 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.588155 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfc4l\" (UniqueName: \"kubernetes.io/projected/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-kube-api-access-kfc4l\") pod \"maas-api-revoke-sub-ae69f747-qh78f\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:32.611619 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:32.611582 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:33.019583 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:33.019385 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f"] Nov 03 10:03:33.027360 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:03:33.027286 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ba376f_eb75_4ecb_b6a0_46c19cc694fa.slice/crio-8db84a97ac2d0f21844e3be177a32eae6318bf65037ae870cec0fddbae0beb15 WatchSource:0}: Error finding container 8db84a97ac2d0f21844e3be177a32eae6318bf65037ae870cec0fddbae0beb15: Status 404 returned error can't find the container with id 8db84a97ac2d0f21844e3be177a32eae6318bf65037ae870cec0fddbae0beb15 Nov 03 10:03:33.612490 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:33.612364 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" event={"ID":"42ba376f-eb75-4ecb-b6a0-46c19cc694fa","Type":"ContainerStarted","Data":"da10a7d773e3f3ee0eeb6b76b54245ee400e3a740c6c0eff65699ce35c1f629e"} Nov 03 10:03:33.612490 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:33.612431 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" event={"ID":"42ba376f-eb75-4ecb-b6a0-46c19cc694fa","Type":"ContainerStarted","Data":"8db84a97ac2d0f21844e3be177a32eae6318bf65037ae870cec0fddbae0beb15"} Nov 03 10:03:34.620221 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:34.620132 5015 generic.go:356] "Generic (PLEG): container finished" podID="42ba376f-eb75-4ecb-b6a0-46c19cc694fa" containerID="da10a7d773e3f3ee0eeb6b76b54245ee400e3a740c6c0eff65699ce35c1f629e" exitCode=0 Nov 03 10:03:34.620647 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:34.620222 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" event={"ID":"42ba376f-eb75-4ecb-b6a0-46c19cc694fa","Type":"ContainerDied","Data":"da10a7d773e3f3ee0eeb6b76b54245ee400e3a740c6c0eff65699ce35c1f629e"} Nov 03 10:03:35.774207 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:35.774179 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:35.906663 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:35.906574 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-openshift-ca\") pod \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " Nov 03 10:03:35.906854 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:35.906680 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-kube-api-access-kfc4l\" (UniqueName: \"kubernetes.io/projected/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-kube-api-access-kfc4l\") pod \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " Nov 03 10:03:35.906854 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:35.906707 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-xks-ca\") pod \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\" (UID: \"42ba376f-eb75-4ecb-b6a0-46c19cc694fa\") " Nov 03 10:03:35.907037 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:35.907007 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-openshift-ca" pod "42ba376f-eb75-4ecb-b6a0-46c19cc694fa" (UID: "42ba376f-eb75-4ecb-b6a0-46c19cc694fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:03:35.908924 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:35.908898 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-xks-ca" pod "42ba376f-eb75-4ecb-b6a0-46c19cc694fa" (UID: "42ba376f-eb75-4ecb-b6a0-46c19cc694fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:03:35.909061 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:35.908973 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-kube-api-access-kfc4l" pod "42ba376f-eb75-4ecb-b6a0-46c19cc694fa" (UID: "42ba376f-eb75-4ecb-b6a0-46c19cc694fa"). InnerVolumeSpecName "kube-api-access-kfc4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:03:36.007741 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:36.007712 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfc4l\" (UniqueName: \"kubernetes.io/projected/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-kube-api-access-kfc4l\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:36.007741 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:36.007739 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:36.007741 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:36.007748 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42ba376f-eb75-4ecb-b6a0-46c19cc694fa-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:36.633858 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:36.633823 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" event={"ID":"42ba376f-eb75-4ecb-b6a0-46c19cc694fa","Type":"ContainerDied","Data":"8db84a97ac2d0f21844e3be177a32eae6318bf65037ae870cec0fddbae0beb15"} Nov 03 10:03:36.634039 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:36.633871 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8db84a97ac2d0f21844e3be177a32eae6318bf65037ae870cec0fddbae0beb15" Nov 03 10:03:36.634039 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:36.633833 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f" Nov 03 10:03:41.784589 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:41.784549 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" podUID="15f6894d-cb7a-45c9-8490-16b8d30c08e1" containerName="main" containerID="cri-o://3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859" gracePeriod=45 Nov 03 10:03:42.055840 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.055818 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:03:42.166567 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.166532 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tls-certs\") pod \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " Nov 03 10:03:42.166742 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.166579 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-model-cache\") pod \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " Nov 03 10:03:42.166742 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.166606 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-home\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-home\") pod \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " Nov 03 10:03:42.166742 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.166623 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tmp-dir\") pod \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " Nov 03 10:03:42.166948 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.166742 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f6894d-cb7a-45c9-8490-16b8d30c08e1-kube-api-access-pb8lg\" (UniqueName: \"kubernetes.io/projected/15f6894d-cb7a-45c9-8490-16b8d30c08e1-kube-api-access-pb8lg\") pod \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " Nov 03 10:03:42.166948 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.166815 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-dshm\") pod \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\" (UID: \"15f6894d-cb7a-45c9-8490-16b8d30c08e1\") " Nov 03 10:03:42.166948 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.166932 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-home" pod "15f6894d-cb7a-45c9-8490-16b8d30c08e1" (UID: "15f6894d-cb7a-45c9-8490-16b8d30c08e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:03:42.167137 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.166947 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-model-cache" pod "15f6894d-cb7a-45c9-8490-16b8d30c08e1" (UID: "15f6894d-cb7a-45c9-8490-16b8d30c08e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:03:42.167248 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.167225 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:42.167337 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.167253 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:42.167337 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.167257 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tmp-dir" pod "15f6894d-cb7a-45c9-8490-16b8d30c08e1" (UID: "15f6894d-cb7a-45c9-8490-16b8d30c08e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:03:42.169765 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.169733 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-dshm" pod "15f6894d-cb7a-45c9-8490-16b8d30c08e1" (UID: "15f6894d-cb7a-45c9-8490-16b8d30c08e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:03:42.169874 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.169787 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tls-certs" pod "15f6894d-cb7a-45c9-8490-16b8d30c08e1" (UID: "15f6894d-cb7a-45c9-8490-16b8d30c08e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:03:42.169874 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.169819 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f6894d-cb7a-45c9-8490-16b8d30c08e1-kube-api-access-pb8lg" pod "15f6894d-cb7a-45c9-8490-16b8d30c08e1" (UID: "15f6894d-cb7a-45c9-8490-16b8d30c08e1"). InnerVolumeSpecName "kube-api-access-pb8lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:03:42.268720 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.268682 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb8lg\" (UniqueName: \"kubernetes.io/projected/15f6894d-cb7a-45c9-8490-16b8d30c08e1-kube-api-access-pb8lg\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:42.268720 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.268715 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:42.268720 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.268724 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:42.268957 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.268733 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15f6894d-cb7a-45c9-8490-16b8d30c08e1-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:03:42.671152 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.671112 5015 generic.go:356] "Generic (PLEG): container finished" podID="15f6894d-cb7a-45c9-8490-16b8d30c08e1" containerID="3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859" exitCode=0 Nov 03 10:03:42.671339 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.671167 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" Nov 03 10:03:42.671339 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.671197 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" event={"ID":"15f6894d-cb7a-45c9-8490-16b8d30c08e1","Type":"ContainerDied","Data":"3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859"} Nov 03 10:03:42.671339 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.671240 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" event={"ID":"15f6894d-cb7a-45c9-8490-16b8d30c08e1","Type":"ContainerDied","Data":"69dd734d717f86301c58c03fbf59ed40b960ddb9170d3d04f00af07e555a65de"} Nov 03 10:03:42.671339 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.671261 5015 scope.go:122] "RemoveContainer" containerID="3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859" Nov 03 10:03:42.681780 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.681759 5015 scope.go:122] "RemoveContainer" containerID="3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859" Nov 03 10:03:42.682062 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:03:42.682041 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859\": container with ID starting with 3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859 not found: ID does not exist" containerID="3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859" Nov 03 10:03:42.682131 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.682075 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859"} err="failed to get container status \"3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859\": rpc error: code = NotFound desc = could not find container \"3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859\": container with ID starting with 3a12efe15b9c91c50648c4a145d84b88b00280a0c464a343d1e1031b87021859 not found: ID does not exist" Nov 03 10:03:42.840654 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.840613 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf"] Nov 03 10:03:42.890850 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:42.890812 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf"] Nov 03 10:03:43.031396 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:43.031344 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-8906966c-kserve-77ffb6fcf8-rgxlf" podUID="15f6894d-cb7a-45c9-8490-16b8d30c08e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.252:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 10:03:43.203491 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:43.203436 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f6894d-cb7a-45c9-8490-16b8d30c08e1" path="/var/lib/kubelet/pods/15f6894d-cb7a-45c9-8490-16b8d30c08e1/volumes" Nov 03 10:03:45.404056 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.404002 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd"] Nov 03 10:03:45.404746 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.404725 5015 cpu_manager.go:414] "containerMap: removing container" podUID="42ba376f-eb75-4ecb-b6a0-46c19cc694fa" containerName="revoke-keys" Nov 03 10:03:45.404828 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.404748 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ba376f-eb75-4ecb-b6a0-46c19cc694fa" containerName="revoke-keys" Nov 03 10:03:45.404828 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.404763 5015 cpu_manager.go:414] "containerMap: removing container" podUID="15f6894d-cb7a-45c9-8490-16b8d30c08e1" containerName="main" Nov 03 10:03:45.404828 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.404771 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f6894d-cb7a-45c9-8490-16b8d30c08e1" containerName="main" Nov 03 10:03:45.404991 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.404899 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ba376f-eb75-4ecb-b6a0-46c19cc694fa" containerName="revoke-keys" Nov 03 10:03:45.404991 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.404914 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f6894d-cb7a-45c9-8490-16b8d30c08e1" containerName="main" Nov 03 10:03:45.409310 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.409285 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.467855 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.467819 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-fd27d2-openshift-default-dockercfg-5c9t7\"" Nov 03 10:03:45.525275 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.525242 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd"] Nov 03 10:03:45.600244 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.600211 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-envoy\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.600453 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.600260 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-podinfo\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.600453 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.600344 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2zql\" (UniqueName: \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-kube-api-access-t2zql\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.600453 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.600447 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-certs\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.600681 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.600560 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-credential-socket\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.600681 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.600579 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-socket\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.600681 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.600600 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef68be5-115b-425d-b4a2-2260fea2b69e-istiod-ca-cert\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.600681 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.600626 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-token\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.600819 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.600720 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-data\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.702323 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.702227 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2zql\" (UniqueName: \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-kube-api-access-t2zql\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.702323 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.702280 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-certs\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.702614 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.702361 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-credential-socket\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.702614 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.702388 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-socket\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.702614 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.702422 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef68be5-115b-425d-b4a2-2260fea2b69e-istiod-ca-cert\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.702614 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.702457 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-token\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.702614 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.702536 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-data\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.702614 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.702595 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-envoy\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.703019 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.702647 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-podinfo\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.703019 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.702756 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-certs\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.703019 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.702913 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-credential-socket\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.703186 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.703129 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-socket\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.703314 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.703272 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef68be5-115b-425d-b4a2-2260fea2b69e-istiod-ca-cert\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.703314 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.703287 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-data\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.705495 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.705448 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-podinfo\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.705722 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.705706 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-envoy\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.761778 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.761750 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-token\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:45.762129 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:45.762110 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2zql\" (UniqueName: \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-kube-api-access-t2zql\") pod \"e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:46.022723 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:46.022684 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:46.454243 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:46.454214 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd"] Nov 03 10:03:46.455987 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:03:46.455964 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef68be5_115b_425d_b4a2_2260fea2b69e.slice/crio-149b9eefee8096d0c43293bfd92c944a45963c2c900fa4ebb8e2c0ab56039645 WatchSource:0}: Error finding container 149b9eefee8096d0c43293bfd92c944a45963c2c900fa4ebb8e2c0ab56039645: Status 404 returned error can't find the container with id 149b9eefee8096d0c43293bfd92c944a45963c2c900fa4ebb8e2c0ab56039645 Nov 03 10:03:46.458247 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:46.458214 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:03:46.458308 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:46.458280 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:03:46.458349 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:46.458309 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:03:46.701613 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:46.701573 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" event={"ID":"3ef68be5-115b-425d-b4a2-2260fea2b69e","Type":"ContainerStarted","Data":"74641e6e251f28a7f6457c69cc67e9f4db5c282d537a89ce52fb168dd62823ec"} Nov 03 10:03:46.701778 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:46.701641 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" event={"ID":"3ef68be5-115b-425d-b4a2-2260fea2b69e","Type":"ContainerStarted","Data":"149b9eefee8096d0c43293bfd92c944a45963c2c900fa4ebb8e2c0ab56039645"} Nov 03 10:03:46.820606 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:46.820498 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" podStartSLOduration=1.820462553 podStartE2EDuration="1.820462553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:03:46.819333612 +0000 UTC m=+5059.744275892" watchObservedRunningTime="2026-11-03 10:03:46.820462553 +0000 UTC m=+5059.745404769" Nov 03 10:03:46.903589 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:03:46.903551 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:03:47.023656 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:47.023621 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:47.028623 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:47.028598 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:47.708255 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:47.708226 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:47.709290 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:47.709272 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:03:53.631379 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.631335 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8"] Nov 03 10:03:53.638662 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.638632 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:03:53.686577 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.686541 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8"] Nov 03 10:03:53.692340 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.692303 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe950fb-7ab9-421c-8237-c4d211f2d94c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8\" (UID: \"1fe950fb-7ab9-421c-8237-c4d211f2d94c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:03:53.692574 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.692363 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmr4r\" (UniqueName: \"kubernetes.io/projected/1fe950fb-7ab9-421c-8237-c4d211f2d94c-kube-api-access-zmr4r\") pod \"payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8\" (UID: \"1fe950fb-7ab9-421c-8237-c4d211f2d94c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:03:53.733609 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.733572 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c0364842-dockercfg-j9g2c\"" Nov 03 10:03:53.733609 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.733612 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-fd27d2\"" Nov 03 10:03:53.793201 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.793170 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe950fb-7ab9-421c-8237-c4d211f2d94c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8\" (UID: \"1fe950fb-7ab9-421c-8237-c4d211f2d94c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:03:53.793426 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.793212 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmr4r\" (UniqueName: \"kubernetes.io/projected/1fe950fb-7ab9-421c-8237-c4d211f2d94c-kube-api-access-zmr4r\") pod \"payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8\" (UID: \"1fe950fb-7ab9-421c-8237-c4d211f2d94c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:03:53.794001 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.793974 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe950fb-7ab9-421c-8237-c4d211f2d94c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8\" (UID: \"1fe950fb-7ab9-421c-8237-c4d211f2d94c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:03:53.807603 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.807578 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8"] Nov 03 10:03:53.811598 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.811581 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:03:53.852525 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.852491 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmr4r\" (UniqueName: \"kubernetes.io/projected/1fe950fb-7ab9-421c-8237-c4d211f2d94c-kube-api-access-zmr4r\") pod \"payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8\" (UID: \"1fe950fb-7ab9-421c-8237-c4d211f2d94c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:03:53.868978 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.868946 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8"] Nov 03 10:03:53.894320 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.894225 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9g56\" (UniqueName: \"kubernetes.io/projected/c6c6ef51-8312-40d4-ab77-7c137dd6455c-kube-api-access-b9g56\") pod \"payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8\" (UID: \"c6c6ef51-8312-40d4-ab77-7c137dd6455c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:03:53.894516 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.894364 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c6ef51-8312-40d4-ab77-7c137dd6455c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8\" (UID: \"c6c6ef51-8312-40d4-ab77-7c137dd6455c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:03:53.953806 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.953773 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:03:53.995413 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.995367 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9g56\" (UniqueName: \"kubernetes.io/projected/c6c6ef51-8312-40d4-ab77-7c137dd6455c-kube-api-access-b9g56\") pod \"payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8\" (UID: \"c6c6ef51-8312-40d4-ab77-7c137dd6455c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:03:53.995632 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.995498 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c6ef51-8312-40d4-ab77-7c137dd6455c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8\" (UID: \"c6c6ef51-8312-40d4-ab77-7c137dd6455c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:03:53.996630 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:53.996422 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c6ef51-8312-40d4-ab77-7c137dd6455c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8\" (UID: \"c6c6ef51-8312-40d4-ab77-7c137dd6455c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:03:54.055378 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.055274 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9g56\" (UniqueName: \"kubernetes.io/projected/c6c6ef51-8312-40d4-ab77-7c137dd6455c-kube-api-access-b9g56\") pod \"payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8\" (UID: \"c6c6ef51-8312-40d4-ab77-7c137dd6455c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:03:54.125028 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.124991 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:03:54.331296 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.330517 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8"] Nov 03 10:03:54.505090 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.505063 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8"] Nov 03 10:03:54.506692 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:03:54.506660 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c6ef51_8312_40d4_ab77_7c137dd6455c.slice/crio-d8651aef990969500050459772eeac17533b85d5f46404c4d643127218d8c61e WatchSource:0}: Error finding container d8651aef990969500050459772eeac17533b85d5f46404c4d643127218d8c61e: Status 404 returned error can't find the container with id d8651aef990969500050459772eeac17533b85d5f46404c4d643127218d8c61e Nov 03 10:03:54.755758 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.755647 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" event={"ID":"c6c6ef51-8312-40d4-ab77-7c137dd6455c","Type":"ContainerStarted","Data":"7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839"} Nov 03 10:03:54.755758 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.755720 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" event={"ID":"c6c6ef51-8312-40d4-ab77-7c137dd6455c","Type":"ContainerStarted","Data":"d8651aef990969500050459772eeac17533b85d5f46404c4d643127218d8c61e"} Nov 03 10:03:54.756272 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.755864 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:03:54.757441 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.757406 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" event={"ID":"1fe950fb-7ab9-421c-8237-c4d211f2d94c","Type":"ContainerStarted","Data":"4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1"} Nov 03 10:03:54.757611 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.757459 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" event={"ID":"1fe950fb-7ab9-421c-8237-c4d211f2d94c","Type":"ContainerStarted","Data":"9432f7b9a693caf74dfce4d4535174d0845b5305bcc8d28704651740fe4cf844"} Nov 03 10:03:54.757611 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.757544 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:03:54.873890 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.873828 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" podStartSLOduration=1.87381116 podStartE2EDuration="1.87381116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:03:54.871236396 +0000 UTC m=+5067.796178614" watchObservedRunningTime="2026-11-03 10:03:54.87381116 +0000 UTC m=+5067.798753414" Nov 03 10:03:54.996630 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:54.996552 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" podStartSLOduration=1.9965302409999999 podStartE2EDuration="1.996530241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:03:54.987848433 +0000 UTC m=+5067.912790652" watchObservedRunningTime="2026-11-03 10:03:54.996530241 +0000 UTC m=+5067.921472461" Nov 03 10:03:56.442630 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.442595 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98"] Nov 03 10:03:56.446842 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.446825 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:56.504656 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.504625 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98"] Nov 03 10:03:56.522238 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.522203 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:56.522439 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.522251 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4bm\" (UniqueName: \"kubernetes.io/projected/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-kube-api-access-rk4bm\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:56.522439 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.522407 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:56.522583 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.522556 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:56.550011 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.549977 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-fd27d2-serving-cert\"" Nov 03 10:03:56.550177 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.550068 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 10:03:56.550226 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.550188 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f42g7\"" Nov 03 10:03:56.623879 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.623845 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4bm\" (UniqueName: \"kubernetes.io/projected/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-kube-api-access-rk4bm\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:56.624079 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.623925 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:56.624079 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.623973 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:56.624079 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.624018 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:56.624217 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:03:56.624081 5015 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-serving-cert: secret "maas-api-e2e-worker-single-main-fd27d2-serving-cert" not found Nov 03 10:03:56.624217 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:03:56.624141 5015 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-maas-api-tls podName:6dfb6d45-5fcd-4c2e-96dc-176affd0cc76 nodeName:}" failed. No retries permitted until 2026-11-03 10:03:57.124126403 +0000 UTC m=+5070.049068598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-maas-api-tls") pod "maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" (UID: "6dfb6d45-5fcd-4c2e-96dc-176affd0cc76") : secret "maas-api-e2e-worker-single-main-fd27d2-serving-cert" not found Nov 03 10:03:56.624796 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.624778 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:56.626725 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.626707 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:56.700712 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:56.700627 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4bm\" (UniqueName: \"kubernetes.io/projected/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-kube-api-access-rk4bm\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:57.129770 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:57.129730 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:57.132381 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:57.132348 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:57.362032 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:57.361998 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:57.738681 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:57.738656 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98"] Nov 03 10:03:57.740588 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:03:57.740563 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfb6d45_5fcd_4c2e_96dc_176affd0cc76.slice/crio-3044a11dda8c65dff59684ae9fda2506a5c0a3b765d92926e7d12753a5ded12a WatchSource:0}: Error finding container 3044a11dda8c65dff59684ae9fda2506a5c0a3b765d92926e7d12753a5ded12a: Status 404 returned error can't find the container with id 3044a11dda8c65dff59684ae9fda2506a5c0a3b765d92926e7d12753a5ded12a Nov 03 10:03:57.742377 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:57.742351 5015 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 10:03:57.778104 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:57.778072 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" event={"ID":"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76","Type":"ContainerStarted","Data":"3044a11dda8c65dff59684ae9fda2506a5c0a3b765d92926e7d12753a5ded12a"} Nov 03 10:03:58.786594 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:58.786547 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" event={"ID":"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76","Type":"ContainerStarted","Data":"932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22"} Nov 03 10:03:58.787006 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:58.786620 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:03:58.946805 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:03:58.946701 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" podStartSLOduration=2.036477494 podStartE2EDuration="2.946685995s" totalImagesPullingTime="910.208501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:03:58.946238598 +0000 UTC m=+5071.871180797" watchObservedRunningTime="2026-11-03 10:03:58.946685995 +0000 UTC m=+5071.871628211" Nov 03 10:04:04.802367 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:04.802333 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:04:05.768689 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:05.768655 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:04:05.770111 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:05.770085 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:04:14.923505 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:14.923433 5015 generic.go:356] "Generic (PLEG): container finished" podID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerID="74641e6e251f28a7f6457c69cc67e9f4db5c282d537a89ce52fb168dd62823ec" exitCode=0 Nov 03 10:04:14.924023 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:14.923512 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" event={"ID":"3ef68be5-115b-425d-b4a2-2260fea2b69e","Type":"ContainerDied","Data":"74641e6e251f28a7f6457c69cc67e9f4db5c282d537a89ce52fb168dd62823ec"} Nov 03 10:04:14.924023 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:14.923921 5015 scope.go:122] "RemoveContainer" containerID="74641e6e251f28a7f6457c69cc67e9f4db5c282d537a89ce52fb168dd62823ec" Nov 03 10:04:14.925156 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:14.925129 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:04:14.925208 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:14.925199 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:04:14.925246 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:14.925231 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:04:15.931554 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:15.931521 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" event={"ID":"3ef68be5-115b-425d-b4a2-2260fea2b69e","Type":"ContainerStarted","Data":"0034ff812b40b76d5924ed57b53de3b93d4e5ece2530cf51fdc3fd0ce098016f"} Nov 03 10:04:16.023264 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:16.023227 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:04:16.024589 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:16.024563 5015 patch_prober.go:29] interesting pod/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Nov 03 10:04:16.024721 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:16.024610 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Nov 03 10:04:16.902936 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:04:16.902897 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:04:17.023771 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:17.023739 5015 patch_prober.go:29] interesting pod/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Nov 03 10:04:17.024163 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:17.023802 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Nov 03 10:04:17.708893 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:17.708857 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:04:18.024045 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:18.024008 5015 patch_prober.go:29] interesting pod/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Nov 03 10:04:18.024446 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:18.024062 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Nov 03 10:04:19.023199 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:19.023163 5015 patch_prober.go:29] interesting pod/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Nov 03 10:04:19.023386 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:19.023216 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Nov 03 10:04:20.024084 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:20.024045 5015 patch_prober.go:29] interesting pod/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Nov 03 10:04:20.024462 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:20.024118 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Nov 03 10:04:21.023724 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:21.023689 5015 patch_prober.go:29] interesting pod/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Nov 03 10:04:21.023902 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:21.023747 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Nov 03 10:04:22.027273 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:22.027234 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:04:22.028368 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:22.028348 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:04:28.159395 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:28.159367 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:04:28.195963 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:28.195935 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:04:46.902117 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:04:46.902079 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:04:56.053242 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:56.053199 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs"] Nov 03 10:04:56.103217 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:56.103176 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rr4bs"] Nov 03 10:04:57.203907 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:04:57.203871 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a3f6eb-9737-4c55-a115-a88d1a8199ad" path="/var/lib/kubelet/pods/92a3f6eb-9737-4c55-a115-a88d1a8199ad/volumes" Nov 03 10:05:16.905678 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:05:16.905631 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:05:29.571814 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:29.571777 5015 scope.go:122] "RemoveContainer" containerID="3b5971a04fa237c2cd31507d421c399ded5b3969b99de3baba48bb4b6817d455" Nov 03 10:05:35.598891 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:35.598805 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8"] Nov 03 10:05:35.710856 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:35.710820 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8"] Nov 03 10:05:36.462827 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:36.462788 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98"] Nov 03 10:05:36.463048 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:36.463025 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" podUID="6dfb6d45-5fcd-4c2e-96dc-176affd0cc76" containerName="maas-api" containerID="cri-o://932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22" gracePeriod=30 Nov 03 10:05:37.732255 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.732222 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:05:37.892054 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.891966 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-kube-api-access-rk4bm\" (UniqueName: \"kubernetes.io/projected/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-kube-api-access-rk4bm\") pod \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " Nov 03 10:05:37.892054 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.892004 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-metrics-certs\") pod \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " Nov 03 10:05:37.892054 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.892042 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-openshift-service-ca\") pod \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " Nov 03 10:05:37.892331 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.892152 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-maas-api-tls\") pod \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\" (UID: \"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76\") " Nov 03 10:05:37.892532 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.892468 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-openshift-service-ca" pod "6dfb6d45-5fcd-4c2e-96dc-176affd0cc76" (UID: "6dfb6d45-5fcd-4c2e-96dc-176affd0cc76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:05:37.892676 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.892623 5015 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-openshift-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:05:37.894469 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.894440 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-metrics-certs" pod "6dfb6d45-5fcd-4c2e-96dc-176affd0cc76" (UID: "6dfb6d45-5fcd-4c2e-96dc-176affd0cc76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:05:37.894610 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.894492 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-maas-api-tls" pod "6dfb6d45-5fcd-4c2e-96dc-176affd0cc76" (UID: "6dfb6d45-5fcd-4c2e-96dc-176affd0cc76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:05:37.894744 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.894726 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-kube-api-access-rk4bm" pod "6dfb6d45-5fcd-4c2e-96dc-176affd0cc76" (UID: "6dfb6d45-5fcd-4c2e-96dc-176affd0cc76"). InnerVolumeSpecName "kube-api-access-rk4bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:05:37.993585 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.993544 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk4bm\" (UniqueName: \"kubernetes.io/projected/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-kube-api-access-rk4bm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:05:37.993585 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.993579 5015 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-metrics-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:05:37.993585 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:37.993589 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76-maas-api-tls\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:05:38.465091 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:38.465052 5015 generic.go:356] "Generic (PLEG): container finished" podID="6dfb6d45-5fcd-4c2e-96dc-176affd0cc76" containerID="932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22" exitCode=0 Nov 03 10:05:38.465283 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:38.465095 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" Nov 03 10:05:38.465283 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:38.465141 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" event={"ID":"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76","Type":"ContainerDied","Data":"932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22"} Nov 03 10:05:38.465283 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:38.465200 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98" event={"ID":"6dfb6d45-5fcd-4c2e-96dc-176affd0cc76","Type":"ContainerDied","Data":"3044a11dda8c65dff59684ae9fda2506a5c0a3b765d92926e7d12753a5ded12a"} Nov 03 10:05:38.465283 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:38.465224 5015 scope.go:122] "RemoveContainer" containerID="932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22" Nov 03 10:05:38.476289 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:38.476261 5015 scope.go:122] "RemoveContainer" containerID="932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22" Nov 03 10:05:38.476639 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:05:38.476616 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22\": container with ID starting with 932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22 not found: ID does not exist" containerID="932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22" Nov 03 10:05:38.476732 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:38.476652 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22"} err="failed to get container status \"932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22\": rpc error: code = NotFound desc = could not find container \"932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22\": container with ID starting with 932f27fb7a5ad6ff74cb1c958d55c70ab53e2423803643cb4207c6725f387c22 not found: ID does not exist" Nov 03 10:05:38.634021 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:38.633982 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98"] Nov 03 10:05:38.684699 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:38.684663 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-tpb98"] Nov 03 10:05:39.203423 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:39.203388 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dfb6d45-5fcd-4c2e-96dc-176affd0cc76" path="/var/lib/kubelet/pods/6dfb6d45-5fcd-4c2e-96dc-176affd0cc76/volumes" Nov 03 10:05:40.600172 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:40.600131 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" podUID="1fe950fb-7ab9-421c-8237-c4d211f2d94c" containerName="payload-processing" containerID="cri-o://4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1" gracePeriod=25 Nov 03 10:05:40.711280 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:40.711238 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" podUID="c6c6ef51-8312-40d4-ab77-7c137dd6455c" containerName="payload-pre-processing" containerID="cri-o://7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839" gracePeriod=25 Nov 03 10:05:41.055301 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:41.055263 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5"] Nov 03 10:05:41.105205 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:41.105171 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-ww8m5"] Nov 03 10:05:41.203519 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:41.203458 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9236d8-ee3b-4fde-9edf-0402b6c8fe21" path="/var/lib/kubelet/pods/9c9236d8-ee3b-4fde-9edf-0402b6c8fe21/volumes" Nov 03 10:05:45.765721 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:45.765684 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" podUID="c6c6ef51-8312-40d4-ab77-7c137dd6455c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:05:45.766201 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:45.765738 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" podUID="1fe950fb-7ab9-421c-8237-c4d211f2d94c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:05:46.904494 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:05:46.904438 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:05:55.766191 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:55.766147 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" podUID="1fe950fb-7ab9-421c-8237-c4d211f2d94c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:05:55.766745 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:05:55.766208 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" podUID="c6c6ef51-8312-40d4-ab77-7c137dd6455c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:06:00.865114 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:00.865043 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:06:00.929691 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:00.929642 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe950fb-7ab9-421c-8237-c4d211f2d94c-kube-api-access-zmr4r\" (UniqueName: \"kubernetes.io/projected/1fe950fb-7ab9-421c-8237-c4d211f2d94c-kube-api-access-zmr4r\") pod \"1fe950fb-7ab9-421c-8237-c4d211f2d94c\" (UID: \"1fe950fb-7ab9-421c-8237-c4d211f2d94c\") " Nov 03 10:06:00.929881 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:00.929844 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe950fb-7ab9-421c-8237-c4d211f2d94c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe950fb-7ab9-421c-8237-c4d211f2d94c-plugins-config-volume\") pod \"1fe950fb-7ab9-421c-8237-c4d211f2d94c\" (UID: \"1fe950fb-7ab9-421c-8237-c4d211f2d94c\") " Nov 03 10:06:00.930305 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:00.930275 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe950fb-7ab9-421c-8237-c4d211f2d94c-plugins-config-volume" pod "1fe950fb-7ab9-421c-8237-c4d211f2d94c" (UID: "1fe950fb-7ab9-421c-8237-c4d211f2d94c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:06:00.932151 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:00.932117 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe950fb-7ab9-421c-8237-c4d211f2d94c-kube-api-access-zmr4r" pod "1fe950fb-7ab9-421c-8237-c4d211f2d94c" (UID: "1fe950fb-7ab9-421c-8237-c4d211f2d94c"). InnerVolumeSpecName "kube-api-access-zmr4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:06:00.953247 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:00.953219 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:06:01.031251 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.031213 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c6ef51-8312-40d4-ab77-7c137dd6455c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c6ef51-8312-40d4-ab77-7c137dd6455c-plugins-config-volume\") pod \"c6c6ef51-8312-40d4-ab77-7c137dd6455c\" (UID: \"c6c6ef51-8312-40d4-ab77-7c137dd6455c\") " Nov 03 10:06:01.031437 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.031266 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c6ef51-8312-40d4-ab77-7c137dd6455c-kube-api-access-b9g56\" (UniqueName: \"kubernetes.io/projected/c6c6ef51-8312-40d4-ab77-7c137dd6455c-kube-api-access-b9g56\") pod \"c6c6ef51-8312-40d4-ab77-7c137dd6455c\" (UID: \"c6c6ef51-8312-40d4-ab77-7c137dd6455c\") " Nov 03 10:06:01.031572 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.031556 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmr4r\" (UniqueName: \"kubernetes.io/projected/1fe950fb-7ab9-421c-8237-c4d211f2d94c-kube-api-access-zmr4r\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:01.031627 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.031574 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe950fb-7ab9-421c-8237-c4d211f2d94c-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:01.031627 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.031595 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c6ef51-8312-40d4-ab77-7c137dd6455c-plugins-config-volume" pod "c6c6ef51-8312-40d4-ab77-7c137dd6455c" (UID: "c6c6ef51-8312-40d4-ab77-7c137dd6455c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:06:01.033917 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.033890 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c6ef51-8312-40d4-ab77-7c137dd6455c-kube-api-access-b9g56" pod "c6c6ef51-8312-40d4-ab77-7c137dd6455c" (UID: "c6c6ef51-8312-40d4-ab77-7c137dd6455c"). InnerVolumeSpecName "kube-api-access-b9g56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:06:01.132623 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.132536 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c6ef51-8312-40d4-ab77-7c137dd6455c-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:01.132623 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.132579 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9g56\" (UniqueName: \"kubernetes.io/projected/c6c6ef51-8312-40d4-ab77-7c137dd6455c-kube-api-access-b9g56\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:01.610566 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.610527 5015 generic.go:356] "Generic (PLEG): container finished" podID="c6c6ef51-8312-40d4-ab77-7c137dd6455c" containerID="7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839" exitCode=0 Nov 03 10:06:01.610776 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.610579 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" Nov 03 10:06:01.610776 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.610626 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" event={"ID":"c6c6ef51-8312-40d4-ab77-7c137dd6455c","Type":"ContainerDied","Data":"7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839"} Nov 03 10:06:01.610776 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.610698 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8" event={"ID":"c6c6ef51-8312-40d4-ab77-7c137dd6455c","Type":"ContainerDied","Data":"d8651aef990969500050459772eeac17533b85d5f46404c4d643127218d8c61e"} Nov 03 10:06:01.610776 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.610730 5015 scope.go:122] "RemoveContainer" containerID="7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839" Nov 03 10:06:01.612157 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.612132 5015 generic.go:356] "Generic (PLEG): container finished" podID="1fe950fb-7ab9-421c-8237-c4d211f2d94c" containerID="4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1" exitCode=0 Nov 03 10:06:01.612246 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.612184 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" Nov 03 10:06:01.612246 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.612225 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" event={"ID":"1fe950fb-7ab9-421c-8237-c4d211f2d94c","Type":"ContainerDied","Data":"4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1"} Nov 03 10:06:01.612333 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.612278 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8" event={"ID":"1fe950fb-7ab9-421c-8237-c4d211f2d94c","Type":"ContainerDied","Data":"9432f7b9a693caf74dfce4d4535174d0845b5305bcc8d28704651740fe4cf844"} Nov 03 10:06:01.628127 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.628093 5015 scope.go:122] "RemoveContainer" containerID="7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839" Nov 03 10:06:01.628562 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:06:01.628522 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839\": container with ID starting with 7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839 not found: ID does not exist" containerID="7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839" Nov 03 10:06:01.628562 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.628550 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839"} err="failed to get container status \"7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839\": rpc error: code = NotFound desc = could not find container \"7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839\": container with ID starting with 7bb0671cf0df4a7ed8e0624d4e42d712452ce51ffa0fe010f197b6d389756839 not found: ID does not exist" Nov 03 10:06:01.628713 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.628568 5015 scope.go:122] "RemoveContainer" containerID="4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1" Nov 03 10:06:01.638009 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.637990 5015 scope.go:122] "RemoveContainer" containerID="4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1" Nov 03 10:06:01.638357 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:06:01.638336 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1\": container with ID starting with 4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1 not found: ID does not exist" containerID="4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1" Nov 03 10:06:01.638398 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.638366 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1"} err="failed to get container status \"4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1\": rpc error: code = NotFound desc = could not find container \"4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1\": container with ID starting with 4edfb255aa05719a4278d59788be6b021cc2e2e2ec951b0e22886247b85d42a1 not found: ID does not exist" Nov 03 10:06:01.780311 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.780275 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8"] Nov 03 10:06:01.830791 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.830755 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fd27d2-54567xqtj8"] Nov 03 10:06:01.899370 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.899287 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8"] Nov 03 10:06:01.903897 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:01.903871 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fd27d2-84cf8fbd89wkx8"] Nov 03 10:06:03.211111 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:03.211076 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe950fb-7ab9-421c-8237-c4d211f2d94c" path="/var/lib/kubelet/pods/1fe950fb-7ab9-421c-8237-c4d211f2d94c/volumes" Nov 03 10:06:03.211512 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:03.211463 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c6ef51-8312-40d4-ab77-7c137dd6455c" path="/var/lib/kubelet/pods/c6c6ef51-8312-40d4-ab77-7c137dd6455c/volumes" Nov 03 10:06:15.053838 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:15.053796 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4"] Nov 03 10:06:15.057303 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:15.057270 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7xsc4"] Nov 03 10:06:15.203560 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:15.203527 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10edf53-264a-4cdf-82e1-6454ec1c7fb0" path="/var/lib/kubelet/pods/f10edf53-264a-4cdf-82e1-6454ec1c7fb0/volumes" Nov 03 10:06:16.902880 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:06:16.902843 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:06:23.986946 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:23.986910 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg"] Nov 03 10:06:23.987350 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:23.987311 5015 cpu_manager.go:414] "containerMap: removing container" podUID="1fe950fb-7ab9-421c-8237-c4d211f2d94c" containerName="payload-processing" Nov 03 10:06:23.987350 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:23.987322 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe950fb-7ab9-421c-8237-c4d211f2d94c" containerName="payload-processing" Nov 03 10:06:23.987350 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:23.987348 5015 cpu_manager.go:414] "containerMap: removing container" podUID="c6c6ef51-8312-40d4-ab77-7c137dd6455c" containerName="payload-pre-processing" Nov 03 10:06:23.987468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:23.987354 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c6ef51-8312-40d4-ab77-7c137dd6455c" containerName="payload-pre-processing" Nov 03 10:06:23.987468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:23.987372 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6dfb6d45-5fcd-4c2e-96dc-176affd0cc76" containerName="maas-api" Nov 03 10:06:23.987468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:23.987378 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dfb6d45-5fcd-4c2e-96dc-176affd0cc76" containerName="maas-api" Nov 03 10:06:23.987468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:23.987444 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c6ef51-8312-40d4-ab77-7c137dd6455c" containerName="payload-pre-processing" Nov 03 10:06:23.987468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:23.987455 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dfb6d45-5fcd-4c2e-96dc-176affd0cc76" containerName="maas-api" Nov 03 10:06:23.987468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:23.987461 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe950fb-7ab9-421c-8237-c4d211f2d94c" containerName="payload-processing" Nov 03 10:06:23.990812 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:23.990796 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:24.048329 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.048297 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg"] Nov 03 10:06:24.094551 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.094520 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:06:24.094726 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.094577 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:06:24.168059 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.168024 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:24.168232 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.168140 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:24.168232 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.168208 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkg8s\" (UniqueName: \"kubernetes.io/projected/08955587-e277-42ae-a571-fa1480a8b7c8-kube-api-access-lkg8s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:24.268956 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.268862 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:24.269147 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.268952 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:24.269147 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.269046 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkg8s\" (UniqueName: \"kubernetes.io/projected/08955587-e277-42ae-a571-fa1480a8b7c8-kube-api-access-lkg8s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:24.269784 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.269719 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:24.271462 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.271433 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:24.328055 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.328018 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkg8s\" (UniqueName: \"kubernetes.io/projected/08955587-e277-42ae-a571-fa1480a8b7c8-kube-api-access-lkg8s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:24.603423 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.603338 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:24.982970 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:24.982942 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg"] Nov 03 10:06:24.984388 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:06:24.984364 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08955587_e277_42ae_a571_fa1480a8b7c8.slice/crio-11055df66df017a2826489ed0688642edc83f6dfd331cdf4acc6d861924e2772 WatchSource:0}: Error finding container 11055df66df017a2826489ed0688642edc83f6dfd331cdf4acc6d861924e2772: Status 404 returned error can't find the container with id 11055df66df017a2826489ed0688642edc83f6dfd331cdf4acc6d861924e2772 Nov 03 10:06:25.760008 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:25.759964 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" event={"ID":"08955587-e277-42ae-a571-fa1480a8b7c8","Type":"ContainerStarted","Data":"b0d3fac74525822d0dc687b86a8f2d4a2311057479b50c44bf9fd54cd8576a3f"} Nov 03 10:06:25.760008 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:25.760015 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" event={"ID":"08955587-e277-42ae-a571-fa1480a8b7c8","Type":"ContainerStarted","Data":"11055df66df017a2826489ed0688642edc83f6dfd331cdf4acc6d861924e2772"} Nov 03 10:06:25.878243 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:25.878188 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" podStartSLOduration=2.878175225 podStartE2EDuration="2.878175225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:06:25.875593829 +0000 UTC m=+5218.800536071" watchObservedRunningTime="2026-11-03 10:06:25.878175225 +0000 UTC m=+5218.803117441" Nov 03 10:06:26.767099 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:26.767066 5015 generic.go:356] "Generic (PLEG): container finished" podID="08955587-e277-42ae-a571-fa1480a8b7c8" containerID="b0d3fac74525822d0dc687b86a8f2d4a2311057479b50c44bf9fd54cd8576a3f" exitCode=0 Nov 03 10:06:26.767099 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:26.767103 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" event={"ID":"08955587-e277-42ae-a571-fa1480a8b7c8","Type":"ContainerDied","Data":"b0d3fac74525822d0dc687b86a8f2d4a2311057479b50c44bf9fd54cd8576a3f"} Nov 03 10:06:27.914465 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:27.914437 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:28.007679 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.007647 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-xks-ca\") pod \"08955587-e277-42ae-a571-fa1480a8b7c8\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " Nov 03 10:06:28.007853 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.007718 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08955587-e277-42ae-a571-fa1480a8b7c8-kube-api-access-lkg8s\" (UniqueName: \"kubernetes.io/projected/08955587-e277-42ae-a571-fa1480a8b7c8-kube-api-access-lkg8s\") pod \"08955587-e277-42ae-a571-fa1480a8b7c8\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " Nov 03 10:06:28.007853 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.007783 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-openshift-ca\") pod \"08955587-e277-42ae-a571-fa1480a8b7c8\" (UID: \"08955587-e277-42ae-a571-fa1480a8b7c8\") " Nov 03 10:06:28.008186 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.008162 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-openshift-ca" pod "08955587-e277-42ae-a571-fa1480a8b7c8" (UID: "08955587-e277-42ae-a571-fa1480a8b7c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:06:28.010021 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.009989 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-xks-ca" pod "08955587-e277-42ae-a571-fa1480a8b7c8" (UID: "08955587-e277-42ae-a571-fa1480a8b7c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:06:28.010141 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.010075 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08955587-e277-42ae-a571-fa1480a8b7c8-kube-api-access-lkg8s" pod "08955587-e277-42ae-a571-fa1480a8b7c8" (UID: "08955587-e277-42ae-a571-fa1480a8b7c8"). InnerVolumeSpecName "kube-api-access-lkg8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:06:28.108766 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.108679 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:28.108766 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.108709 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08955587-e277-42ae-a571-fa1480a8b7c8-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:28.108766 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.108720 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkg8s\" (UniqueName: \"kubernetes.io/projected/08955587-e277-42ae-a571-fa1480a8b7c8-kube-api-access-lkg8s\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:28.781654 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.781623 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" Nov 03 10:06:28.781843 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.781622 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg" event={"ID":"08955587-e277-42ae-a571-fa1480a8b7c8","Type":"ContainerDied","Data":"11055df66df017a2826489ed0688642edc83f6dfd331cdf4acc6d861924e2772"} Nov 03 10:06:28.781843 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:28.781748 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11055df66df017a2826489ed0688642edc83f6dfd331cdf4acc6d861924e2772" Nov 03 10:06:29.599508 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:29.599460 5015 scope.go:122] "RemoveContainer" containerID="15d6c8524929d7e57917a1dcefbc1cbdd57a9565db058447eedcea2ef5fb0999" Nov 03 10:06:35.052893 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:35.052858 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595"] Nov 03 10:06:35.103203 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:35.103172 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dp595"] Nov 03 10:06:35.203255 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:35.203219 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a66ec3-51b9-425e-82e9-210cce020726" path="/var/lib/kubelet/pods/50a66ec3-51b9-425e-82e9-210cce020726/volumes" Nov 03 10:06:44.548401 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:44.548359 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg"] Nov 03 10:06:44.598754 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:44.598718 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fd27d2-vj8wg"] Nov 03 10:06:44.762690 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:44.762659 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd"] Nov 03 10:06:44.762951 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:44.762921 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" containerID="cri-o://0034ff812b40b76d5924ed57b53de3b93d4e5ece2530cf51fdc3fd0ce098016f" gracePeriod=30 Nov 03 10:06:45.203653 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:45.203620 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08955587-e277-42ae-a571-fa1480a8b7c8" path="/var/lib/kubelet/pods/08955587-e277-42ae-a571-fa1480a8b7c8/volumes" Nov 03 10:06:46.903520 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:06:46.903444 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:06:47.052271 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:47.052236 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr"] Nov 03 10:06:47.102410 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:47.102374 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mdbpr"] Nov 03 10:06:47.203806 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:47.203728 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e27578-79ff-4c2e-868d-f954d653c665" path="/var/lib/kubelet/pods/f3e27578-79ff-4c2e-868d-f954d653c665/volumes" Nov 03 10:06:49.914893 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:49.914859 5015 generic.go:356] "Generic (PLEG): container finished" podID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerID="0034ff812b40b76d5924ed57b53de3b93d4e5ece2530cf51fdc3fd0ce098016f" exitCode=0 Nov 03 10:06:49.915317 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:49.914932 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" event={"ID":"3ef68be5-115b-425d-b4a2-2260fea2b69e","Type":"ContainerDied","Data":"0034ff812b40b76d5924ed57b53de3b93d4e5ece2530cf51fdc3fd0ce098016f"} Nov 03 10:06:49.915317 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:49.915012 5015 scope.go:122] "RemoveContainer" containerID="74641e6e251f28a7f6457c69cc67e9f4db5c282d537a89ce52fb168dd62823ec" Nov 03 10:06:50.036554 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.036467 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:06:50.127614 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.127584 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-data\") pod \"3ef68be5-115b-425d-b4a2-2260fea2b69e\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " Nov 03 10:06:50.127799 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.127648 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-podinfo\") pod \"3ef68be5-115b-425d-b4a2-2260fea2b69e\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " Nov 03 10:06:50.127799 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.127732 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-certs\") pod \"3ef68be5-115b-425d-b4a2-2260fea2b69e\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " Nov 03 10:06:50.127799 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.127792 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef68be5-115b-425d-b4a2-2260fea2b69e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef68be5-115b-425d-b4a2-2260fea2b69e-istiod-ca-cert\") pod \"3ef68be5-115b-425d-b4a2-2260fea2b69e\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " Nov 03 10:06:50.127981 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.127828 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-kube-api-access-t2zql\" (UniqueName: \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-kube-api-access-t2zql\") pod \"3ef68be5-115b-425d-b4a2-2260fea2b69e\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " Nov 03 10:06:50.127981 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.127855 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-token\" (UniqueName: \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-token\") pod \"3ef68be5-115b-425d-b4a2-2260fea2b69e\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " Nov 03 10:06:50.127981 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.127877 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-data" pod "3ef68be5-115b-425d-b4a2-2260fea2b69e" (UID: "3ef68be5-115b-425d-b4a2-2260fea2b69e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:06:50.127981 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.127908 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-credential-socket\") pod \"3ef68be5-115b-425d-b4a2-2260fea2b69e\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " Nov 03 10:06:50.127981 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.127946 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-envoy\") pod \"3ef68be5-115b-425d-b4a2-2260fea2b69e\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " Nov 03 10:06:50.127981 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.127973 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-socket\") pod \"3ef68be5-115b-425d-b4a2-2260fea2b69e\" (UID: \"3ef68be5-115b-425d-b4a2-2260fea2b69e\") " Nov 03 10:06:50.128308 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.128125 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-certs" pod "3ef68be5-115b-425d-b4a2-2260fea2b69e" (UID: "3ef68be5-115b-425d-b4a2-2260fea2b69e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:06:50.128308 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.128275 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-credential-socket" pod "3ef68be5-115b-425d-b4a2-2260fea2b69e" (UID: "3ef68be5-115b-425d-b4a2-2260fea2b69e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:06:50.128432 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.128299 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef68be5-115b-425d-b4a2-2260fea2b69e-istiod-ca-cert" pod "3ef68be5-115b-425d-b4a2-2260fea2b69e" (UID: "3ef68be5-115b-425d-b4a2-2260fea2b69e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:06:50.128432 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.128398 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:50.128432 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.128423 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef68be5-115b-425d-b4a2-2260fea2b69e-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:50.128432 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.128425 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-socket" pod "3ef68be5-115b-425d-b4a2-2260fea2b69e" (UID: "3ef68be5-115b-425d-b4a2-2260fea2b69e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:06:50.128660 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.128441 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:50.128660 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.128455 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:50.130400 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.130374 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-podinfo" pod "3ef68be5-115b-425d-b4a2-2260fea2b69e" (UID: "3ef68be5-115b-425d-b4a2-2260fea2b69e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:06:50.130543 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.130372 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-envoy" pod "3ef68be5-115b-425d-b4a2-2260fea2b69e" (UID: "3ef68be5-115b-425d-b4a2-2260fea2b69e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:06:50.130807 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.130780 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-kube-api-access-t2zql" pod "3ef68be5-115b-425d-b4a2-2260fea2b69e" (UID: "3ef68be5-115b-425d-b4a2-2260fea2b69e"). InnerVolumeSpecName "kube-api-access-t2zql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:06:50.130807 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.130792 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-token" pod "3ef68be5-115b-425d-b4a2-2260fea2b69e" (UID: "3ef68be5-115b-425d-b4a2-2260fea2b69e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:06:50.229183 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.229143 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2zql\" (UniqueName: \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-kube-api-access-t2zql\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:50.229183 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.229183 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:50.229382 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.229199 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:50.229382 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.229215 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef68be5-115b-425d-b4a2-2260fea2b69e-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:50.229382 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.229229 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef68be5-115b-425d-b4a2-2260fea2b69e-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:50.921904 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.921871 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" Nov 03 10:06:50.922339 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.921903 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd" event={"ID":"3ef68be5-115b-425d-b4a2-2260fea2b69e","Type":"ContainerDied","Data":"149b9eefee8096d0c43293bfd92c944a45963c2c900fa4ebb8e2c0ab56039645"} Nov 03 10:06:50.922339 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:50.921961 5015 scope.go:122] "RemoveContainer" containerID="0034ff812b40b76d5924ed57b53de3b93d4e5ece2530cf51fdc3fd0ce098016f" Nov 03 10:06:51.091795 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:51.091753 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd"] Nov 03 10:06:51.143234 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:51.143202 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fd27d2-openshift-default-88cfdb6bf-p6vbd"] Nov 03 10:06:51.204174 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:06:51.204087 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" path="/var/lib/kubelet/pods/3ef68be5-115b-425d-b4a2-2260fea2b69e/volumes" Nov 03 10:07:04.177043 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.177012 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96"] Nov 03 10:07:04.177502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.177408 5015 cpu_manager.go:414] "containerMap: removing container" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" Nov 03 10:07:04.177502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.177418 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" Nov 03 10:07:04.177502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.177430 5015 cpu_manager.go:414] "containerMap: removing container" podUID="08955587-e277-42ae-a571-fa1480a8b7c8" containerName="revoke-keys" Nov 03 10:07:04.177502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.177435 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08955587-e277-42ae-a571-fa1480a8b7c8" containerName="revoke-keys" Nov 03 10:07:04.177502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.177443 5015 cpu_manager.go:414] "containerMap: removing container" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" Nov 03 10:07:04.177502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.177447 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" Nov 03 10:07:04.177724 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.177547 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" Nov 03 10:07:04.177724 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.177555 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="08955587-e277-42ae-a571-fa1480a8b7c8" containerName="revoke-keys" Nov 03 10:07:04.177724 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.177561 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef68be5-115b-425d-b4a2-2260fea2b69e" containerName="istio-proxy" Nov 03 10:07:04.182094 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.182061 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.239920 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.239880 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96"] Nov 03 10:07:04.286564 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.286535 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f8cb05bc-kserve-self-signed-certs\"" Nov 03 10:07:04.367327 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.367293 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-model-cache\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.367327 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.367331 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0124b6f9-7814-4dbc-8c30-481cb7972b86-tls-certs\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.367590 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.367415 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6fh8\" (UniqueName: \"kubernetes.io/projected/0124b6f9-7814-4dbc-8c30-481cb7972b86-kube-api-access-n6fh8\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.367590 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.367446 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-home\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.367671 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.367590 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-tmp-dir\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.367671 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.367614 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-dshm\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.468187 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.468109 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-tmp-dir\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.468187 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.468143 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-dshm\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.468416 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.468205 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-model-cache\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.468416 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.468221 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0124b6f9-7814-4dbc-8c30-481cb7972b86-tls-certs\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.468416 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.468251 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6fh8\" (UniqueName: \"kubernetes.io/projected/0124b6f9-7814-4dbc-8c30-481cb7972b86-kube-api-access-n6fh8\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.468416 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.468276 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-home\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.468674 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.468585 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-tmp-dir\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.468674 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.468644 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-model-cache\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.468756 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.468738 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-home\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.470762 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.470735 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-dshm\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.471011 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.470988 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0124b6f9-7814-4dbc-8c30-481cb7972b86-tls-certs\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.527147 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.527113 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6fh8\" (UniqueName: \"kubernetes.io/projected/0124b6f9-7814-4dbc-8c30-481cb7972b86-kube-api-access-n6fh8\") pod \"e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:04.794459 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:04.794428 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:05.171133 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:05.171104 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96"] Nov 03 10:07:05.173026 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:07:05.172989 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0124b6f9_7814_4dbc_8c30_481cb7972b86.slice/crio-07cdddb1c2afb523d4e7c6dd44bc25971a22c14ed52ef62341fb3e84390db4b5 WatchSource:0}: Error finding container 07cdddb1c2afb523d4e7c6dd44bc25971a22c14ed52ef62341fb3e84390db4b5: Status 404 returned error can't find the container with id 07cdddb1c2afb523d4e7c6dd44bc25971a22c14ed52ef62341fb3e84390db4b5 Nov 03 10:07:06.024319 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:06.024269 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" event={"ID":"0124b6f9-7814-4dbc-8c30-481cb7972b86","Type":"ContainerStarted","Data":"00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383"} Nov 03 10:07:06.024843 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:06.024327 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" event={"ID":"0124b6f9-7814-4dbc-8c30-481cb7972b86","Type":"ContainerStarted","Data":"07cdddb1c2afb523d4e7c6dd44bc25971a22c14ed52ef62341fb3e84390db4b5"} Nov 03 10:07:06.142727 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:06.142678 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" podStartSLOduration=2.142664722 podStartE2EDuration="2.142664722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:07:06.140789518 +0000 UTC m=+5259.065731751" watchObservedRunningTime="2026-11-03 10:07:06.142664722 +0000 UTC m=+5259.067606938" Nov 03 10:07:06.795189 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:06.795143 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:14.795265 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:14.795232 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:14.807534 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:14.807508 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:15.154998 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:15.154917 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:07:16.901712 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:07:16.901672 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:07:29.635234 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:29.635200 5015 scope.go:122] "RemoveContainer" containerID="d767d0298938972c4f9e53a90b319d963c5470d63847ba2c2bd036e65e85863e" Nov 03 10:07:29.644745 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:29.644724 5015 scope.go:122] "RemoveContainer" containerID="1949e673b05e6ad443e70c2ad38bc4e0485c582e37ed67bf0b3a48217d23653d" Nov 03 10:07:36.007745 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.007713 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj"] Nov 03 10:07:36.012722 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.012702 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.101023 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.100990 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628c76cb-a465-4e8b-83f9-1bd03113095e-tls-certs\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.101231 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.101033 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-tmp-dir\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.101231 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.101066 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-dshm\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.101361 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.101267 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-home\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.101361 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.101327 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4gnh\" (UniqueName: \"kubernetes.io/projected/628c76cb-a465-4e8b-83f9-1bd03113095e-kube-api-access-x4gnh\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.101465 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.101413 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-model-cache\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.110266 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.110227 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj"] Nov 03 10:07:36.129832 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.129811 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f8cb05bc-kserve-self-signed-certs\"" Nov 03 10:07:36.202697 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.202665 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-home\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.202874 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.202704 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gnh\" (UniqueName: \"kubernetes.io/projected/628c76cb-a465-4e8b-83f9-1bd03113095e-kube-api-access-x4gnh\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.202874 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.202748 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-model-cache\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.202951 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.202897 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628c76cb-a465-4e8b-83f9-1bd03113095e-tls-certs\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.202997 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.202951 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-tmp-dir\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.203102 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.203078 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-dshm\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.203102 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.203093 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-home\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.203231 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.203164 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-model-cache\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.203294 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.203233 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-tmp-dir\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.205308 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.205287 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-dshm\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.205666 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.205648 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628c76cb-a465-4e8b-83f9-1bd03113095e-tls-certs\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.261952 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.261892 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4gnh\" (UniqueName: \"kubernetes.io/projected/628c76cb-a465-4e8b-83f9-1bd03113095e-kube-api-access-x4gnh\") pod \"e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.325705 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.325678 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:36.721001 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:07:36.720971 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod628c76cb_a465_4e8b_83f9_1bd03113095e.slice/crio-e153eb3db43bfebb410a2835849c9d857d08cf4d8a3f610e72ac679a4dc9ac17 WatchSource:0}: Error finding container e153eb3db43bfebb410a2835849c9d857d08cf4d8a3f610e72ac679a4dc9ac17: Status 404 returned error can't find the container with id e153eb3db43bfebb410a2835849c9d857d08cf4d8a3f610e72ac679a4dc9ac17 Nov 03 10:07:36.721574 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:36.721542 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj"] Nov 03 10:07:37.280367 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:37.280330 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" event={"ID":"628c76cb-a465-4e8b-83f9-1bd03113095e","Type":"ContainerStarted","Data":"a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33"} Nov 03 10:07:37.280812 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:37.280381 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" event={"ID":"628c76cb-a465-4e8b-83f9-1bd03113095e","Type":"ContainerStarted","Data":"e153eb3db43bfebb410a2835849c9d857d08cf4d8a3f610e72ac679a4dc9ac17"} Nov 03 10:07:37.326881 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:37.326853 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:37.397039 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:37.396988 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" podStartSLOduration=2.396974519 podStartE2EDuration="2.396974519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:07:37.396876567 +0000 UTC m=+5290.321818787" watchObservedRunningTime="2026-11-03 10:07:37.396974519 +0000 UTC m=+5290.321916735" Nov 03 10:07:46.326357 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:46.326317 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:46.339734 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:46.339705 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:46.357239 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:07:46.357211 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:07:46.901248 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:07:46.901212 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:08:08.426420 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.426377 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc"] Nov 03 10:08:08.430146 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.430124 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:08.487970 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.487936 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc"] Nov 03 10:08:08.533810 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.533773 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:08:08.534003 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.533841 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:08:08.565523 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.565457 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2rrqc\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:08.565705 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.565673 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2rrqc\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:08.565763 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.565733 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46p4x\" (UniqueName: \"kubernetes.io/projected/411954b6-8724-4754-afde-8cc9c2008d4f-kube-api-access-46p4x\") pod \"maas-api-revoke-sub-452b6d7b-2rrqc\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:08.667033 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.666977 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46p4x\" (UniqueName: \"kubernetes.io/projected/411954b6-8724-4754-afde-8cc9c2008d4f-kube-api-access-46p4x\") pod \"maas-api-revoke-sub-452b6d7b-2rrqc\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:08.667033 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.667046 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2rrqc\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:08.667275 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.667127 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2rrqc\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:08.667788 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.667767 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2rrqc\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:08.669821 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.669800 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2rrqc\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:08.726844 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.726816 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46p4x\" (UniqueName: \"kubernetes.io/projected/411954b6-8724-4754-afde-8cc9c2008d4f-kube-api-access-46p4x\") pod \"maas-api-revoke-sub-452b6d7b-2rrqc\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:08.745665 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:08.745625 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:09.126137 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:09.126108 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc"] Nov 03 10:08:09.127155 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:08:09.127130 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411954b6_8724_4754_afde_8cc9c2008d4f.slice/crio-50e93ab36871647802ec04c117dc11b5f6e9186e8b73acbe141238bcc8e27ed8 WatchSource:0}: Error finding container 50e93ab36871647802ec04c117dc11b5f6e9186e8b73acbe141238bcc8e27ed8: Status 404 returned error can't find the container with id 50e93ab36871647802ec04c117dc11b5f6e9186e8b73acbe141238bcc8e27ed8 Nov 03 10:08:09.496413 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:09.496363 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" event={"ID":"411954b6-8724-4754-afde-8cc9c2008d4f","Type":"ContainerStarted","Data":"d5b1c21a6114f6325022cc5912bf30ebdf98a0071eab6a987c321d7ae7813492"} Nov 03 10:08:09.496910 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:09.496434 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" event={"ID":"411954b6-8724-4754-afde-8cc9c2008d4f","Type":"ContainerStarted","Data":"50e93ab36871647802ec04c117dc11b5f6e9186e8b73acbe141238bcc8e27ed8"} Nov 03 10:08:09.612722 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:09.612665 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" podStartSLOduration=1.612649971 podStartE2EDuration="1.612649971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:08:09.611785009 +0000 UTC m=+5322.536727249" watchObservedRunningTime="2026-11-03 10:08:09.612649971 +0000 UTC m=+5322.537592181" Nov 03 10:08:10.503912 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:10.503866 5015 generic.go:356] "Generic (PLEG): container finished" podID="411954b6-8724-4754-afde-8cc9c2008d4f" containerID="d5b1c21a6114f6325022cc5912bf30ebdf98a0071eab6a987c321d7ae7813492" exitCode=0 Nov 03 10:08:10.504296 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:10.503943 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" event={"ID":"411954b6-8724-4754-afde-8cc9c2008d4f","Type":"ContainerDied","Data":"d5b1c21a6114f6325022cc5912bf30ebdf98a0071eab6a987c321d7ae7813492"} Nov 03 10:08:11.734151 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:11.734121 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:11.802562 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:11.801815 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-openshift-ca\") pod \"411954b6-8724-4754-afde-8cc9c2008d4f\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " Nov 03 10:08:11.802562 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:11.801892 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411954b6-8724-4754-afde-8cc9c2008d4f-kube-api-access-46p4x\" (UniqueName: \"kubernetes.io/projected/411954b6-8724-4754-afde-8cc9c2008d4f-kube-api-access-46p4x\") pod \"411954b6-8724-4754-afde-8cc9c2008d4f\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " Nov 03 10:08:11.802562 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:11.801971 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-xks-ca\") pod \"411954b6-8724-4754-afde-8cc9c2008d4f\" (UID: \"411954b6-8724-4754-afde-8cc9c2008d4f\") " Nov 03 10:08:11.802562 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:11.802217 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-openshift-ca" pod "411954b6-8724-4754-afde-8cc9c2008d4f" (UID: "411954b6-8724-4754-afde-8cc9c2008d4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:08:11.802562 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:11.802422 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:11.805460 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:11.805367 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-xks-ca" pod "411954b6-8724-4754-afde-8cc9c2008d4f" (UID: "411954b6-8724-4754-afde-8cc9c2008d4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:08:11.805460 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:11.805396 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411954b6-8724-4754-afde-8cc9c2008d4f-kube-api-access-46p4x" pod "411954b6-8724-4754-afde-8cc9c2008d4f" (UID: "411954b6-8724-4754-afde-8cc9c2008d4f"). InnerVolumeSpecName "kube-api-access-46p4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:08:11.907690 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:11.904189 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46p4x\" (UniqueName: \"kubernetes.io/projected/411954b6-8724-4754-afde-8cc9c2008d4f-kube-api-access-46p4x\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:11.907690 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:11.904243 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411954b6-8724-4754-afde-8cc9c2008d4f-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:12.523765 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:12.523731 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" Nov 03 10:08:12.523982 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:12.523777 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc" event={"ID":"411954b6-8724-4754-afde-8cc9c2008d4f","Type":"ContainerDied","Data":"50e93ab36871647802ec04c117dc11b5f6e9186e8b73acbe141238bcc8e27ed8"} Nov 03 10:08:12.523982 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:12.523822 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50e93ab36871647802ec04c117dc11b5f6e9186e8b73acbe141238bcc8e27ed8" Nov 03 10:08:16.901961 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:08:16.901920 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:08:17.052632 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:17.052589 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p"] Nov 03 10:08:17.103417 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:17.103380 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5549abbc-rsx9p"] Nov 03 10:08:17.203371 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:17.203293 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82dfe9be-d230-490b-8805-e0fc32069592" path="/var/lib/kubelet/pods/82dfe9be-d230-490b-8805-e0fc32069592/volumes" Nov 03 10:08:21.297327 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:21.297289 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96"] Nov 03 10:08:29.676898 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:29.676865 5015 scope.go:122] "RemoveContainer" containerID="1ff0b6502c251e73c836336ed609fa2a32f56b8b225b0edef6f96f1a24559762" Nov 03 10:08:29.686739 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:29.686715 5015 scope.go:122] "RemoveContainer" containerID="622e2bc20d285c4a0cda126b86c2f136b44f2adf2dddefb197de12788c66fc41" Nov 03 10:08:36.052312 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.052227 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f"] Nov 03 10:08:36.102144 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.102103 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae69f747-qh78f"] Nov 03 10:08:36.318298 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.318204 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" podUID="0124b6f9-7814-4dbc-8c30-481cb7972b86" containerName="main" containerID="cri-o://00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383" gracePeriod=45 Nov 03 10:08:36.586855 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.586827 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:08:36.690367 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.690329 5015 generic.go:356] "Generic (PLEG): container finished" podID="0124b6f9-7814-4dbc-8c30-481cb7972b86" containerID="00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383" exitCode=0 Nov 03 10:08:36.690569 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.690390 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" Nov 03 10:08:36.690569 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.690416 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" event={"ID":"0124b6f9-7814-4dbc-8c30-481cb7972b86","Type":"ContainerDied","Data":"00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383"} Nov 03 10:08:36.690569 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.690462 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96" event={"ID":"0124b6f9-7814-4dbc-8c30-481cb7972b86","Type":"ContainerDied","Data":"07cdddb1c2afb523d4e7c6dd44bc25971a22c14ed52ef62341fb3e84390db4b5"} Nov 03 10:08:36.690569 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.690502 5015 scope.go:122] "RemoveContainer" containerID="00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383" Nov 03 10:08:36.702766 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.702740 5015 scope.go:122] "RemoveContainer" containerID="00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383" Nov 03 10:08:36.703373 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.703054 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-dshm\") pod \"0124b6f9-7814-4dbc-8c30-481cb7972b86\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " Nov 03 10:08:36.703594 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.703570 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-home\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-home\") pod \"0124b6f9-7814-4dbc-8c30-481cb7972b86\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " Nov 03 10:08:36.703708 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.703622 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-tmp-dir\") pod \"0124b6f9-7814-4dbc-8c30-481cb7972b86\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " Nov 03 10:08:36.703708 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.703653 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0124b6f9-7814-4dbc-8c30-481cb7972b86-kube-api-access-n6fh8\" (UniqueName: \"kubernetes.io/projected/0124b6f9-7814-4dbc-8c30-481cb7972b86-kube-api-access-n6fh8\") pod \"0124b6f9-7814-4dbc-8c30-481cb7972b86\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " Nov 03 10:08:36.703828 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.703770 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-model-cache\") pod \"0124b6f9-7814-4dbc-8c30-481cb7972b86\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " Nov 03 10:08:36.703828 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.703800 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0124b6f9-7814-4dbc-8c30-481cb7972b86-tls-certs\" (UniqueName: \"kubernetes.io/secret/0124b6f9-7814-4dbc-8c30-481cb7972b86-tls-certs\") pod \"0124b6f9-7814-4dbc-8c30-481cb7972b86\" (UID: \"0124b6f9-7814-4dbc-8c30-481cb7972b86\") " Nov 03 10:08:36.705605 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.704871 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-home" pod "0124b6f9-7814-4dbc-8c30-481cb7972b86" (UID: "0124b6f9-7814-4dbc-8c30-481cb7972b86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:08:36.705605 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.705246 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-model-cache" pod "0124b6f9-7814-4dbc-8c30-481cb7972b86" (UID: "0124b6f9-7814-4dbc-8c30-481cb7972b86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:08:36.707905 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.705895 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-tmp-dir" pod "0124b6f9-7814-4dbc-8c30-481cb7972b86" (UID: "0124b6f9-7814-4dbc-8c30-481cb7972b86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:08:36.707905 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.706268 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-dshm" pod "0124b6f9-7814-4dbc-8c30-481cb7972b86" (UID: "0124b6f9-7814-4dbc-8c30-481cb7972b86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:08:36.707905 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.708463 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0124b6f9-7814-4dbc-8c30-481cb7972b86-kube-api-access-n6fh8" pod "0124b6f9-7814-4dbc-8c30-481cb7972b86" (UID: "0124b6f9-7814-4dbc-8c30-481cb7972b86"). InnerVolumeSpecName "kube-api-access-n6fh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:08:36.707905 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:08:36.709327 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383\": container with ID starting with 00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383 not found: ID does not exist" containerID="00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383" Nov 03 10:08:36.707905 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.709366 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383"} err="failed to get container status \"00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383\": rpc error: code = NotFound desc = could not find container \"00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383\": container with ID starting with 00877942e6eb234ab342795005d769932a14a5773c0269f3c9c861c77683e383 not found: ID does not exist" Nov 03 10:08:36.713063 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.713039 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0124b6f9-7814-4dbc-8c30-481cb7972b86-tls-certs" pod "0124b6f9-7814-4dbc-8c30-481cb7972b86" (UID: "0124b6f9-7814-4dbc-8c30-481cb7972b86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:08:36.805153 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.805116 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:36.805153 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.805149 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:36.805389 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.805180 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:36.805389 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.805193 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6fh8\" (UniqueName: \"kubernetes.io/projected/0124b6f9-7814-4dbc-8c30-481cb7972b86-kube-api-access-n6fh8\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:36.805389 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.805203 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0124b6f9-7814-4dbc-8c30-481cb7972b86-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:36.805389 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:36.805213 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0124b6f9-7814-4dbc-8c30-481cb7972b86-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:37.114300 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:37.114259 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96"] Nov 03 10:08:37.164594 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:37.164549 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f8cb05bc-kserve-6f954c55b9-nlj96"] Nov 03 10:08:37.204156 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:37.204125 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0124b6f9-7814-4dbc-8c30-481cb7972b86" path="/var/lib/kubelet/pods/0124b6f9-7814-4dbc-8c30-481cb7972b86/volumes" Nov 03 10:08:37.204549 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:37.204535 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ba376f-eb75-4ecb-b6a0-46c19cc694fa" path="/var/lib/kubelet/pods/42ba376f-eb75-4ecb-b6a0-46c19cc694fa/volumes" Nov 03 10:08:38.194662 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:38.194626 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj"] Nov 03 10:08:42.384741 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.384699 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9"] Nov 03 10:08:42.385402 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.385380 5015 cpu_manager.go:414] "containerMap: removing container" podUID="411954b6-8724-4754-afde-8cc9c2008d4f" containerName="revoke-keys" Nov 03 10:08:42.385448 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.385406 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411954b6-8724-4754-afde-8cc9c2008d4f" containerName="revoke-keys" Nov 03 10:08:42.385448 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.385442 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0124b6f9-7814-4dbc-8c30-481cb7972b86" containerName="main" Nov 03 10:08:42.385568 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.385451 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0124b6f9-7814-4dbc-8c30-481cb7972b86" containerName="main" Nov 03 10:08:42.385622 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.385608 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0124b6f9-7814-4dbc-8c30-481cb7972b86" containerName="main" Nov 03 10:08:42.385657 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.385635 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="411954b6-8724-4754-afde-8cc9c2008d4f" containerName="revoke-keys" Nov 03 10:08:42.389105 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.389080 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:42.444678 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.444638 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9"] Nov 03 10:08:42.467844 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.467798 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-x74z9\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:42.468070 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.467872 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-x74z9\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:42.468135 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.468055 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjccg\" (UniqueName: \"kubernetes.io/projected/739c6d98-468c-4a34-9820-0ca7a7efdc8c-kube-api-access-wjccg\") pod \"maas-api-revoke-sub-3e31a506-x74z9\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:42.490741 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.490711 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:08:42.490922 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.490771 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:08:42.568832 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.568729 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-x74z9\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:42.568832 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.568793 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-x74z9\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:42.569096 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.568865 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjccg\" (UniqueName: \"kubernetes.io/projected/739c6d98-468c-4a34-9820-0ca7a7efdc8c-kube-api-access-wjccg\") pod \"maas-api-revoke-sub-3e31a506-x74z9\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:42.569418 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.569395 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-x74z9\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:42.571586 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.571554 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-x74z9\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:42.628541 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.628506 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjccg\" (UniqueName: \"kubernetes.io/projected/739c6d98-468c-4a34-9820-0ca7a7efdc8c-kube-api-access-wjccg\") pod \"maas-api-revoke-sub-3e31a506-x74z9\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:42.703078 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:42.703044 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:43.081673 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:43.081642 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9"] Nov 03 10:08:43.082542 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:08:43.082515 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739c6d98_468c_4a34_9820_0ca7a7efdc8c.slice/crio-2ff2f8dc99944b236f1a870dd953c52719a2ca412a8a1cedf7b2522162b6e0f2 WatchSource:0}: Error finding container 2ff2f8dc99944b236f1a870dd953c52719a2ca412a8a1cedf7b2522162b6e0f2: Status 404 returned error can't find the container with id 2ff2f8dc99944b236f1a870dd953c52719a2ca412a8a1cedf7b2522162b6e0f2 Nov 03 10:08:43.741156 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:43.741117 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" event={"ID":"739c6d98-468c-4a34-9820-0ca7a7efdc8c","Type":"ContainerStarted","Data":"6ebff90b5d71113f960a15c4c0fa94823fe239d58f600440cef8631acb1fad6c"} Nov 03 10:08:43.741760 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:43.741187 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" event={"ID":"739c6d98-468c-4a34-9820-0ca7a7efdc8c","Type":"ContainerStarted","Data":"2ff2f8dc99944b236f1a870dd953c52719a2ca412a8a1cedf7b2522162b6e0f2"} Nov 03 10:08:43.858363 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:43.858284 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" podStartSLOduration=1.858263029 podStartE2EDuration="1.858263029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:08:43.855747156 +0000 UTC m=+5356.780689400" watchObservedRunningTime="2026-11-03 10:08:43.858263029 +0000 UTC m=+5356.783205247" Nov 03 10:08:44.804801 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:44.804759 5015 generic.go:356] "Generic (PLEG): container finished" podID="739c6d98-468c-4a34-9820-0ca7a7efdc8c" containerID="6ebff90b5d71113f960a15c4c0fa94823fe239d58f600440cef8631acb1fad6c" exitCode=0 Nov 03 10:08:44.805546 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:44.805468 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" event={"ID":"739c6d98-468c-4a34-9820-0ca7a7efdc8c","Type":"ContainerDied","Data":"6ebff90b5d71113f960a15c4c0fa94823fe239d58f600440cef8631acb1fad6c"} Nov 03 10:08:45.508673 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:45.508624 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:08:45.508921 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:45.508691 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:08:45.833942 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:45.833728 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:08:45.833942 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:45.833798 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:08:46.017925 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.017899 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:46.028817 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.028786 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-xks-ca\") pod \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " Nov 03 10:08:46.028992 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.028826 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739c6d98-468c-4a34-9820-0ca7a7efdc8c-kube-api-access-wjccg\" (UniqueName: \"kubernetes.io/projected/739c6d98-468c-4a34-9820-0ca7a7efdc8c-kube-api-access-wjccg\") pod \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " Nov 03 10:08:46.028992 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.028856 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-openshift-ca\") pod \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\" (UID: \"739c6d98-468c-4a34-9820-0ca7a7efdc8c\") " Nov 03 10:08:46.029227 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.029200 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-openshift-ca" pod "739c6d98-468c-4a34-9820-0ca7a7efdc8c" (UID: "739c6d98-468c-4a34-9820-0ca7a7efdc8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:08:46.031175 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.031151 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-xks-ca" pod "739c6d98-468c-4a34-9820-0ca7a7efdc8c" (UID: "739c6d98-468c-4a34-9820-0ca7a7efdc8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:08:46.031271 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.031231 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739c6d98-468c-4a34-9820-0ca7a7efdc8c-kube-api-access-wjccg" pod "739c6d98-468c-4a34-9820-0ca7a7efdc8c" (UID: "739c6d98-468c-4a34-9820-0ca7a7efdc8c"). InnerVolumeSpecName "kube-api-access-wjccg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:08:46.130020 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.129925 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:46.130020 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.129963 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjccg\" (UniqueName: \"kubernetes.io/projected/739c6d98-468c-4a34-9820-0ca7a7efdc8c-kube-api-access-wjccg\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:46.130020 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.129978 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739c6d98-468c-4a34-9820-0ca7a7efdc8c-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:46.824013 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.823980 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" Nov 03 10:08:46.824217 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.824026 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9" event={"ID":"739c6d98-468c-4a34-9820-0ca7a7efdc8c","Type":"ContainerDied","Data":"2ff2f8dc99944b236f1a870dd953c52719a2ca412a8a1cedf7b2522162b6e0f2"} Nov 03 10:08:46.824217 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:46.824084 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ff2f8dc99944b236f1a870dd953c52719a2ca412a8a1cedf7b2522162b6e0f2" Nov 03 10:08:46.903190 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:08:46.903153 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:08:53.215556 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.215512 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" podUID="628c76cb-a465-4e8b-83f9-1bd03113095e" containerName="main" containerID="cri-o://a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33" gracePeriod=45 Nov 03 10:08:53.347324 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.347274 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" podUID="628c76cb-a465-4e8b-83f9-1bd03113095e" containerName="main" probeResult="failure" output="Get \"https://10.132.1.5:8000/ready\": dial tcp 10.132.1.5:8000: connect: connection refused" Nov 03 10:08:53.483599 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.483570 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:08:53.498723 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.498689 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-tmp-dir\") pod \"628c76cb-a465-4e8b-83f9-1bd03113095e\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " Nov 03 10:08:53.498923 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.498770 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-home\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-home\") pod \"628c76cb-a465-4e8b-83f9-1bd03113095e\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " Nov 03 10:08:53.498923 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.498802 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-dshm\") pod \"628c76cb-a465-4e8b-83f9-1bd03113095e\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " Nov 03 10:08:53.498923 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.498828 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-model-cache\") pod \"628c76cb-a465-4e8b-83f9-1bd03113095e\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " Nov 03 10:08:53.498923 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.498884 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/628c76cb-a465-4e8b-83f9-1bd03113095e-tls-certs\" (UniqueName: \"kubernetes.io/secret/628c76cb-a465-4e8b-83f9-1bd03113095e-tls-certs\") pod \"628c76cb-a465-4e8b-83f9-1bd03113095e\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " Nov 03 10:08:53.499151 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.498927 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628c76cb-a465-4e8b-83f9-1bd03113095e-kube-api-access-x4gnh\" (UniqueName: \"kubernetes.io/projected/628c76cb-a465-4e8b-83f9-1bd03113095e-kube-api-access-x4gnh\") pod \"628c76cb-a465-4e8b-83f9-1bd03113095e\" (UID: \"628c76cb-a465-4e8b-83f9-1bd03113095e\") " Nov 03 10:08:53.499151 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.499022 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-tmp-dir" pod "628c76cb-a465-4e8b-83f9-1bd03113095e" (UID: "628c76cb-a465-4e8b-83f9-1bd03113095e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:08:53.499151 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.499080 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-home" pod "628c76cb-a465-4e8b-83f9-1bd03113095e" (UID: "628c76cb-a465-4e8b-83f9-1bd03113095e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:08:53.499253 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.499178 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-model-cache" pod "628c76cb-a465-4e8b-83f9-1bd03113095e" (UID: "628c76cb-a465-4e8b-83f9-1bd03113095e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:08:53.499287 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.499244 5015 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-tmp-dir\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:53.499287 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.499265 5015 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-home\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:53.501347 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.501315 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-dshm" pod "628c76cb-a465-4e8b-83f9-1bd03113095e" (UID: "628c76cb-a465-4e8b-83f9-1bd03113095e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:08:53.501537 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.501409 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/628c76cb-a465-4e8b-83f9-1bd03113095e-tls-certs" pod "628c76cb-a465-4e8b-83f9-1bd03113095e" (UID: "628c76cb-a465-4e8b-83f9-1bd03113095e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:08:53.501627 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.501529 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628c76cb-a465-4e8b-83f9-1bd03113095e-kube-api-access-x4gnh" pod "628c76cb-a465-4e8b-83f9-1bd03113095e" (UID: "628c76cb-a465-4e8b-83f9-1bd03113095e"). InnerVolumeSpecName "kube-api-access-x4gnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:08:53.600317 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.599914 5015 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-dshm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:53.600317 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.599952 5015 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628c76cb-a465-4e8b-83f9-1bd03113095e-model-cache\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:53.600317 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.599967 5015 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628c76cb-a465-4e8b-83f9-1bd03113095e-tls-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:53.600317 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.599982 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4gnh\" (UniqueName: \"kubernetes.io/projected/628c76cb-a465-4e8b-83f9-1bd03113095e-kube-api-access-x4gnh\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:08:53.869727 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.869689 5015 generic.go:356] "Generic (PLEG): container finished" podID="628c76cb-a465-4e8b-83f9-1bd03113095e" containerID="a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33" exitCode=0 Nov 03 10:08:53.869933 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.869737 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" Nov 03 10:08:53.869933 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.869778 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" event={"ID":"628c76cb-a465-4e8b-83f9-1bd03113095e","Type":"ContainerDied","Data":"a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33"} Nov 03 10:08:53.869933 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.869828 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj" event={"ID":"628c76cb-a465-4e8b-83f9-1bd03113095e","Type":"ContainerDied","Data":"e153eb3db43bfebb410a2835849c9d857d08cf4d8a3f610e72ac679a4dc9ac17"} Nov 03 10:08:53.869933 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.869849 5015 scope.go:122] "RemoveContainer" containerID="a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33" Nov 03 10:08:53.880143 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.880124 5015 scope.go:122] "RemoveContainer" containerID="a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33" Nov 03 10:08:53.880445 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:08:53.880427 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33\": container with ID starting with a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33 not found: ID does not exist" containerID="a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33" Nov 03 10:08:53.880557 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:53.880460 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33"} err="failed to get container status \"a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33\": rpc error: code = NotFound desc = could not find container \"a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33\": container with ID starting with a1fd3e47879a9ae41b97a0effedf842addd7cc057c22df612abf22e26313ae33 not found: ID does not exist" Nov 03 10:08:54.038696 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:54.038661 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj"] Nov 03 10:08:54.089158 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:54.089119 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f8cb05bc-kserve-6896b67d7c-mflbj"] Nov 03 10:08:55.203996 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:08:55.203955 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="628c76cb-a465-4e8b-83f9-1bd03113095e" path="/var/lib/kubelet/pods/628c76cb-a465-4e8b-83f9-1bd03113095e/volumes" Nov 03 10:09:07.054312 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.054270 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq"] Nov 03 10:09:07.055032 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.054926 5015 cpu_manager.go:414] "containerMap: removing container" podUID="628c76cb-a465-4e8b-83f9-1bd03113095e" containerName="main" Nov 03 10:09:07.055032 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.054945 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628c76cb-a465-4e8b-83f9-1bd03113095e" containerName="main" Nov 03 10:09:07.055032 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.054958 5015 cpu_manager.go:414] "containerMap: removing container" podUID="739c6d98-468c-4a34-9820-0ca7a7efdc8c" containerName="revoke-keys" Nov 03 10:09:07.055032 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.054963 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739c6d98-468c-4a34-9820-0ca7a7efdc8c" containerName="revoke-keys" Nov 03 10:09:07.055301 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.055055 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="628c76cb-a465-4e8b-83f9-1bd03113095e" containerName="main" Nov 03 10:09:07.055301 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.055065 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="739c6d98-468c-4a34-9820-0ca7a7efdc8c" containerName="revoke-keys" Nov 03 10:09:07.059130 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.059108 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:07.116231 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.116190 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq"] Nov 03 10:09:07.162750 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.162706 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:09:07.162942 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.162876 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:09:07.163022 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.162999 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d42e7a-nwfsq\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:07.163066 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.163048 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d42e7a-nwfsq\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:07.163156 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.163139 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbgnf\" (UniqueName: \"kubernetes.io/projected/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-kube-api-access-qbgnf\") pod \"maas-api-revoke-sub-88d42e7a-nwfsq\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:07.264594 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.264555 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d42e7a-nwfsq\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:07.264824 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.264614 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbgnf\" (UniqueName: \"kubernetes.io/projected/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-kube-api-access-qbgnf\") pod \"maas-api-revoke-sub-88d42e7a-nwfsq\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:07.264824 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.264688 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d42e7a-nwfsq\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:07.265327 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.265305 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d42e7a-nwfsq\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:07.267171 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.267146 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d42e7a-nwfsq\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:07.324217 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.324144 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbgnf\" (UniqueName: \"kubernetes.io/projected/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-kube-api-access-qbgnf\") pod \"maas-api-revoke-sub-88d42e7a-nwfsq\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:07.373257 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.373212 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:07.750383 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:07.750355 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq"] Nov 03 10:09:07.751541 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:09:07.751512 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3f113a_bc81_4013_aca8_cd6d474b6a9c.slice/crio-d34a8abe1e07e47cab26b638fff5ba710ef41b29017519265587a513f57bef7c WatchSource:0}: Error finding container d34a8abe1e07e47cab26b638fff5ba710ef41b29017519265587a513f57bef7c: Status 404 returned error can't find the container with id d34a8abe1e07e47cab26b638fff5ba710ef41b29017519265587a513f57bef7c Nov 03 10:09:08.031576 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:08.031467 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" event={"ID":"2c3f113a-bc81-4013-aca8-cd6d474b6a9c","Type":"ContainerStarted","Data":"d34a8abe1e07e47cab26b638fff5ba710ef41b29017519265587a513f57bef7c"} Nov 03 10:09:09.040034 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:09.039995 5015 generic.go:356] "Generic (PLEG): container finished" podID="2c3f113a-bc81-4013-aca8-cd6d474b6a9c" containerID="3f9fb32e286f9de7a9649d4dd5754d888ba6b3e3fd309c8d253f598b056a71c0" exitCode=0 Nov 03 10:09:09.040464 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:09.040092 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" event={"ID":"2c3f113a-bc81-4013-aca8-cd6d474b6a9c","Type":"ContainerDied","Data":"3f9fb32e286f9de7a9649d4dd5754d888ba6b3e3fd309c8d253f598b056a71c0"} Nov 03 10:09:10.187180 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:10.187153 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:10.295929 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:10.295897 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-openshift-ca\") pod \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " Nov 03 10:09:10.296154 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:10.295973 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-xks-ca\") pod \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " Nov 03 10:09:10.296154 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:10.296018 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-kube-api-access-qbgnf\" (UniqueName: \"kubernetes.io/projected/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-kube-api-access-qbgnf\") pod \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\" (UID: \"2c3f113a-bc81-4013-aca8-cd6d474b6a9c\") " Nov 03 10:09:10.296457 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:10.296407 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-openshift-ca" pod "2c3f113a-bc81-4013-aca8-cd6d474b6a9c" (UID: "2c3f113a-bc81-4013-aca8-cd6d474b6a9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:09:10.298971 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:10.298908 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-kube-api-access-qbgnf" pod "2c3f113a-bc81-4013-aca8-cd6d474b6a9c" (UID: "2c3f113a-bc81-4013-aca8-cd6d474b6a9c"). InnerVolumeSpecName "kube-api-access-qbgnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:09:10.299121 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:10.298989 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-xks-ca" pod "2c3f113a-bc81-4013-aca8-cd6d474b6a9c" (UID: "2c3f113a-bc81-4013-aca8-cd6d474b6a9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:09:10.397464 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:10.397417 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:09:10.397464 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:10.397456 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:09:10.397464 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:10.397471 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbgnf\" (UniqueName: \"kubernetes.io/projected/2c3f113a-bc81-4013-aca8-cd6d474b6a9c-kube-api-access-qbgnf\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:09:11.053699 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:11.053668 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" Nov 03 10:09:11.053907 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:11.053667 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq" event={"ID":"2c3f113a-bc81-4013-aca8-cd6d474b6a9c","Type":"ContainerDied","Data":"d34a8abe1e07e47cab26b638fff5ba710ef41b29017519265587a513f57bef7c"} Nov 03 10:09:11.053907 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:11.053801 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d34a8abe1e07e47cab26b638fff5ba710ef41b29017519265587a513f57bef7c" Nov 03 10:09:16.902038 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:09:16.902002 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:09:28.212101 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:28.212067 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:09:28.242104 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:28.242078 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:09:29.715263 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:29.715232 5015 scope.go:122] "RemoveContainer" containerID="8fcffbfa3ac194766e1124968e31b3690deba3131acc4106a7d2de7dc68caee8" Nov 03 10:09:33.802274 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:33.802236 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz"] Nov 03 10:09:33.802730 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:33.802715 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2c3f113a-bc81-4013-aca8-cd6d474b6a9c" containerName="revoke-keys" Nov 03 10:09:33.802778 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:33.802732 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3f113a-bc81-4013-aca8-cd6d474b6a9c" containerName="revoke-keys" Nov 03 10:09:33.802854 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:33.802840 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3f113a-bc81-4013-aca8-cd6d474b6a9c" containerName="revoke-keys" Nov 03 10:09:33.807367 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:33.807350 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:09:33.863619 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:33.863585 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz"] Nov 03 10:09:33.910650 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:33.910609 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-c7jcr\"" Nov 03 10:09:33.910650 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:33.910609 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 10:09:33.928655 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:33.928621 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf4hb\" (UniqueName: \"kubernetes.io/projected/62def2be-9634-4eea-b41a-ae568cd233f8-kube-api-access-rf4hb\") pod \"payload-pre-processing-6c5d968f79-4qzmz\" (UID: \"62def2be-9634-4eea-b41a-ae568cd233f8\") " pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:09:33.928861 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:33.928793 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62def2be-9634-4eea-b41a-ae568cd233f8-plugins-config-volume\") pod \"payload-pre-processing-6c5d968f79-4qzmz\" (UID: \"62def2be-9634-4eea-b41a-ae568cd233f8\") " pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:09:34.029366 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:34.029334 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62def2be-9634-4eea-b41a-ae568cd233f8-plugins-config-volume\") pod \"payload-pre-processing-6c5d968f79-4qzmz\" (UID: \"62def2be-9634-4eea-b41a-ae568cd233f8\") " pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:09:34.029366 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:34.029378 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf4hb\" (UniqueName: \"kubernetes.io/projected/62def2be-9634-4eea-b41a-ae568cd233f8-kube-api-access-rf4hb\") pod \"payload-pre-processing-6c5d968f79-4qzmz\" (UID: \"62def2be-9634-4eea-b41a-ae568cd233f8\") " pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:09:34.030153 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:34.030133 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62def2be-9634-4eea-b41a-ae568cd233f8-plugins-config-volume\") pod \"payload-pre-processing-6c5d968f79-4qzmz\" (UID: \"62def2be-9634-4eea-b41a-ae568cd233f8\") " pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:09:34.089741 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:34.089661 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf4hb\" (UniqueName: \"kubernetes.io/projected/62def2be-9634-4eea-b41a-ae568cd233f8-kube-api-access-rf4hb\") pod \"payload-pre-processing-6c5d968f79-4qzmz\" (UID: \"62def2be-9634-4eea-b41a-ae568cd233f8\") " pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:09:34.120679 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:34.120635 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:09:34.498507 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:34.498448 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz"] Nov 03 10:09:34.499690 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:09:34.499656 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62def2be_9634_4eea_b41a_ae568cd233f8.slice/crio-f18521705e260a51bfd21d7d7b119d535680e7ae79794a550b8f0b8b155992c2 WatchSource:0}: Error finding container f18521705e260a51bfd21d7d7b119d535680e7ae79794a550b8f0b8b155992c2: Status 404 returned error can't find the container with id f18521705e260a51bfd21d7d7b119d535680e7ae79794a550b8f0b8b155992c2 Nov 03 10:09:35.203642 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:35.203601 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" event={"ID":"62def2be-9634-4eea-b41a-ae568cd233f8","Type":"ContainerStarted","Data":"d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e"} Nov 03 10:09:35.204038 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:35.203659 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" event={"ID":"62def2be-9634-4eea-b41a-ae568cd233f8","Type":"ContainerStarted","Data":"f18521705e260a51bfd21d7d7b119d535680e7ae79794a550b8f0b8b155992c2"} Nov 03 10:09:35.204038 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:35.203688 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:09:35.319903 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:35.319853 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" podStartSLOduration=2.319837175 podStartE2EDuration="2.319837175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:09:35.318720136 +0000 UTC m=+5408.243662378" watchObservedRunningTime="2026-11-03 10:09:35.319837175 +0000 UTC m=+5408.244779393" Nov 03 10:09:46.211987 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:46.211950 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:09:46.902376 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:09:46.902336 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:09:48.377579 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:48.377542 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5976d7f594-zrfzg"] Nov 03 10:09:48.381314 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:48.381297 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:09:48.437213 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:48.437176 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5976d7f594-zrfzg"] Nov 03 10:09:48.468700 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:48.468654 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ac979f-120f-4676-90a3-130a107f727f-plugins-config-volume\") pod \"payload-processing-5976d7f594-zrfzg\" (UID: \"77ac979f-120f-4676-90a3-130a107f727f\") " pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:09:48.468909 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:48.468726 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r88gm\" (UniqueName: \"kubernetes.io/projected/77ac979f-120f-4676-90a3-130a107f727f-kube-api-access-r88gm\") pod \"payload-processing-5976d7f594-zrfzg\" (UID: \"77ac979f-120f-4676-90a3-130a107f727f\") " pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:09:48.569880 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:48.569840 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ac979f-120f-4676-90a3-130a107f727f-plugins-config-volume\") pod \"payload-processing-5976d7f594-zrfzg\" (UID: \"77ac979f-120f-4676-90a3-130a107f727f\") " pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:09:48.570093 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:48.569902 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r88gm\" (UniqueName: \"kubernetes.io/projected/77ac979f-120f-4676-90a3-130a107f727f-kube-api-access-r88gm\") pod \"payload-processing-5976d7f594-zrfzg\" (UID: \"77ac979f-120f-4676-90a3-130a107f727f\") " pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:09:48.570583 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:48.570562 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ac979f-120f-4676-90a3-130a107f727f-plugins-config-volume\") pod \"payload-processing-5976d7f594-zrfzg\" (UID: \"77ac979f-120f-4676-90a3-130a107f727f\") " pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:09:48.630277 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:48.630208 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r88gm\" (UniqueName: \"kubernetes.io/projected/77ac979f-120f-4676-90a3-130a107f727f-kube-api-access-r88gm\") pod \"payload-processing-5976d7f594-zrfzg\" (UID: \"77ac979f-120f-4676-90a3-130a107f727f\") " pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:09:48.695855 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:48.695820 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:09:49.073944 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:49.073911 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5976d7f594-zrfzg"] Nov 03 10:09:49.075606 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:09:49.075577 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ac979f_120f_4676_90a3_130a107f727f.slice/crio-cada38c76a359c8c677b13f196aace318336eec50aca58cd0736c57148db10e4 WatchSource:0}: Error finding container cada38c76a359c8c677b13f196aace318336eec50aca58cd0736c57148db10e4: Status 404 returned error can't find the container with id cada38c76a359c8c677b13f196aace318336eec50aca58cd0736c57148db10e4 Nov 03 10:09:49.286647 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:49.286606 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" event={"ID":"77ac979f-120f-4676-90a3-130a107f727f","Type":"ContainerStarted","Data":"72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56"} Nov 03 10:09:49.286853 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:49.286669 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" event={"ID":"77ac979f-120f-4676-90a3-130a107f727f","Type":"ContainerStarted","Data":"cada38c76a359c8c677b13f196aace318336eec50aca58cd0736c57148db10e4"} Nov 03 10:09:49.286853 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:49.286775 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:09:49.403712 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:09:49.403606 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" podStartSLOduration=1.403590777 podStartE2EDuration="1.403590777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:09:49.402616266 +0000 UTC m=+5422.327558502" watchObservedRunningTime="2026-11-03 10:09:49.403590777 +0000 UTC m=+5422.328532993" Nov 03 10:10:00.295082 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:00.295052 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:10:02.628890 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.628850 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-85f55897bb-jwz7j"] Nov 03 10:10:02.635234 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.635206 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" Nov 03 10:10:02.684822 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.684781 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85f55897bb-jwz7j"] Nov 03 10:10:02.715347 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.715312 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-669cd\" (UniqueName: \"kubernetes.io/projected/8b000dd7-72c1-491d-8500-6acde8c672b0-kube-api-access-669cd\") pod \"payload-processing-85f55897bb-jwz7j\" (UID: \"8b000dd7-72c1-491d-8500-6acde8c672b0\") " pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" Nov 03 10:10:02.715541 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.715387 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b000dd7-72c1-491d-8500-6acde8c672b0-plugins-config-volume\") pod \"payload-processing-85f55897bb-jwz7j\" (UID: \"8b000dd7-72c1-491d-8500-6acde8c672b0\") " pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" Nov 03 10:10:02.796819 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.796778 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw"] Nov 03 10:10:02.800574 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.800557 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" Nov 03 10:10:02.816343 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.816317 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-669cd\" (UniqueName: \"kubernetes.io/projected/8b000dd7-72c1-491d-8500-6acde8c672b0-kube-api-access-669cd\") pod \"payload-processing-85f55897bb-jwz7j\" (UID: \"8b000dd7-72c1-491d-8500-6acde8c672b0\") " pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" Nov 03 10:10:02.816493 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.816381 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b000dd7-72c1-491d-8500-6acde8c672b0-plugins-config-volume\") pod \"payload-processing-85f55897bb-jwz7j\" (UID: \"8b000dd7-72c1-491d-8500-6acde8c672b0\") " pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" Nov 03 10:10:02.817091 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.817072 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b000dd7-72c1-491d-8500-6acde8c672b0-plugins-config-volume\") pod \"payload-processing-85f55897bb-jwz7j\" (UID: \"8b000dd7-72c1-491d-8500-6acde8c672b0\") " pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" Nov 03 10:10:02.858554 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.858514 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw"] Nov 03 10:10:02.875534 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.875505 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-669cd\" (UniqueName: \"kubernetes.io/projected/8b000dd7-72c1-491d-8500-6acde8c672b0-kube-api-access-669cd\") pod \"payload-processing-85f55897bb-jwz7j\" (UID: \"8b000dd7-72c1-491d-8500-6acde8c672b0\") " pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" Nov 03 10:10:02.917713 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.917637 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97n9d\" (UniqueName: \"kubernetes.io/projected/6d2b9c62-2c04-43b1-a529-cfb16c231ecd-kube-api-access-97n9d\") pod \"payload-pre-processing-6cf9dd8f69-nc6dw\" (UID: \"6d2b9c62-2c04-43b1-a529-cfb16c231ecd\") " pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" Nov 03 10:10:02.917855 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.917760 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2b9c62-2c04-43b1-a529-cfb16c231ecd-plugins-config-volume\") pod \"payload-pre-processing-6cf9dd8f69-nc6dw\" (UID: \"6d2b9c62-2c04-43b1-a529-cfb16c231ecd\") " pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" Nov 03 10:10:02.948957 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:02.948927 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" Nov 03 10:10:03.021997 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:03.021959 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97n9d\" (UniqueName: \"kubernetes.io/projected/6d2b9c62-2c04-43b1-a529-cfb16c231ecd-kube-api-access-97n9d\") pod \"payload-pre-processing-6cf9dd8f69-nc6dw\" (UID: \"6d2b9c62-2c04-43b1-a529-cfb16c231ecd\") " pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" Nov 03 10:10:03.022165 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:03.022042 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2b9c62-2c04-43b1-a529-cfb16c231ecd-plugins-config-volume\") pod \"payload-pre-processing-6cf9dd8f69-nc6dw\" (UID: \"6d2b9c62-2c04-43b1-a529-cfb16c231ecd\") " pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" Nov 03 10:10:03.022741 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:03.022718 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2b9c62-2c04-43b1-a529-cfb16c231ecd-plugins-config-volume\") pod \"payload-pre-processing-6cf9dd8f69-nc6dw\" (UID: \"6d2b9c62-2c04-43b1-a529-cfb16c231ecd\") " pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" Nov 03 10:10:03.082306 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:03.082270 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97n9d\" (UniqueName: \"kubernetes.io/projected/6d2b9c62-2c04-43b1-a529-cfb16c231ecd-kube-api-access-97n9d\") pod \"payload-pre-processing-6cf9dd8f69-nc6dw\" (UID: \"6d2b9c62-2c04-43b1-a529-cfb16c231ecd\") " pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" Nov 03 10:10:03.113177 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:03.113142 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" Nov 03 10:10:03.324362 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:03.324326 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85f55897bb-jwz7j"] Nov 03 10:10:03.325767 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:10:03.325742 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b000dd7_72c1_491d_8500_6acde8c672b0.slice/crio-2a938e0d09647cc9eb9f0ca59a1ab5af5f2f2d5b35857d1d9011deb0bb46de14 WatchSource:0}: Error finding container 2a938e0d09647cc9eb9f0ca59a1ab5af5f2f2d5b35857d1d9011deb0bb46de14: Status 404 returned error can't find the container with id 2a938e0d09647cc9eb9f0ca59a1ab5af5f2f2d5b35857d1d9011deb0bb46de14 Nov 03 10:10:03.371618 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:03.371408 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" event={"ID":"8b000dd7-72c1-491d-8500-6acde8c672b0","Type":"ContainerStarted","Data":"2a938e0d09647cc9eb9f0ca59a1ab5af5f2f2d5b35857d1d9011deb0bb46de14"} Nov 03 10:10:03.491614 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:03.491583 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw"] Nov 03 10:10:03.492789 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:10:03.492763 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2b9c62_2c04_43b1_a529_cfb16c231ecd.slice/crio-85213d5a2ebc333c2201e01c8d60ea0526afc601306024080909f1c591a45faa WatchSource:0}: Error finding container 85213d5a2ebc333c2201e01c8d60ea0526afc601306024080909f1c591a45faa: Status 404 returned error can't find the container with id 85213d5a2ebc333c2201e01c8d60ea0526afc601306024080909f1c591a45faa Nov 03 10:10:04.379364 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:04.379327 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" event={"ID":"6d2b9c62-2c04-43b1-a529-cfb16c231ecd","Type":"ContainerStarted","Data":"9be16cb735bde72425fdba8f2e12955489258dc26ec741ef0d3767262177d9ce"} Nov 03 10:10:04.379931 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:04.379387 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" event={"ID":"6d2b9c62-2c04-43b1-a529-cfb16c231ecd","Type":"ContainerStarted","Data":"85213d5a2ebc333c2201e01c8d60ea0526afc601306024080909f1c591a45faa"} Nov 03 10:10:04.379931 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:04.379528 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" Nov 03 10:10:04.381288 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:04.381255 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" event={"ID":"8b000dd7-72c1-491d-8500-6acde8c672b0","Type":"ContainerStarted","Data":"9e6e96227f9479956ef76447267b591f03dda9c9258c68a6032c694f5ca9ee6d"} Nov 03 10:10:04.381431 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:04.381337 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" Nov 03 10:10:04.500439 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:04.500376 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" podStartSLOduration=2.500360568 podStartE2EDuration="2.500360568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:10:04.496648231 +0000 UTC m=+5437.421590448" watchObservedRunningTime="2026-11-03 10:10:04.500360568 +0000 UTC m=+5437.425302763" Nov 03 10:10:04.615030 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:04.614965 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" podStartSLOduration=2.614949111 podStartE2EDuration="2.614949111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:10:04.611981515 +0000 UTC m=+5437.536923762" watchObservedRunningTime="2026-11-03 10:10:04.614949111 +0000 UTC m=+5437.539891348" Nov 03 10:10:15.391781 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:15.391700 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-85f55897bb-jwz7j" Nov 03 10:10:15.391781 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:15.391780 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6cf9dd8f69-nc6dw" Nov 03 10:10:15.561554 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:15.561516 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz"] Nov 03 10:10:15.651871 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:15.651769 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5976d7f594-zrfzg"] Nov 03 10:10:16.902226 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:10:16.902193 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:10:20.562430 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:20.562385 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" podUID="62def2be-9634-4eea-b41a-ae568cd233f8" containerName="payload-pre-processing" containerID="cri-o://d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e" gracePeriod=25 Nov 03 10:10:20.652163 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:20.652120 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" podUID="77ac979f-120f-4676-90a3-130a107f727f" containerName="payload-processing" containerID="cri-o://72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56" gracePeriod=25 Nov 03 10:10:20.921157 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:20.921133 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:10:20.947866 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:20.947844 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:10:21.004159 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.004128 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62def2be-9634-4eea-b41a-ae568cd233f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62def2be-9634-4eea-b41a-ae568cd233f8-plugins-config-volume\") pod \"62def2be-9634-4eea-b41a-ae568cd233f8\" (UID: \"62def2be-9634-4eea-b41a-ae568cd233f8\") " Nov 03 10:10:21.004329 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.004232 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62def2be-9634-4eea-b41a-ae568cd233f8-kube-api-access-rf4hb\" (UniqueName: \"kubernetes.io/projected/62def2be-9634-4eea-b41a-ae568cd233f8-kube-api-access-rf4hb\") pod \"62def2be-9634-4eea-b41a-ae568cd233f8\" (UID: \"62def2be-9634-4eea-b41a-ae568cd233f8\") " Nov 03 10:10:21.004630 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.004598 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62def2be-9634-4eea-b41a-ae568cd233f8-plugins-config-volume" pod "62def2be-9634-4eea-b41a-ae568cd233f8" (UID: "62def2be-9634-4eea-b41a-ae568cd233f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:10:21.004773 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.004690 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62def2be-9634-4eea-b41a-ae568cd233f8-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:10:21.006469 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.006446 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62def2be-9634-4eea-b41a-ae568cd233f8-kube-api-access-rf4hb" pod "62def2be-9634-4eea-b41a-ae568cd233f8" (UID: "62def2be-9634-4eea-b41a-ae568cd233f8"). InnerVolumeSpecName "kube-api-access-rf4hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:10:21.105194 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.105119 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ac979f-120f-4676-90a3-130a107f727f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ac979f-120f-4676-90a3-130a107f727f-plugins-config-volume\") pod \"77ac979f-120f-4676-90a3-130a107f727f\" (UID: \"77ac979f-120f-4676-90a3-130a107f727f\") " Nov 03 10:10:21.105351 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.105278 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ac979f-120f-4676-90a3-130a107f727f-kube-api-access-r88gm\" (UniqueName: \"kubernetes.io/projected/77ac979f-120f-4676-90a3-130a107f727f-kube-api-access-r88gm\") pod \"77ac979f-120f-4676-90a3-130a107f727f\" (UID: \"77ac979f-120f-4676-90a3-130a107f727f\") " Nov 03 10:10:21.105511 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.105467 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf4hb\" (UniqueName: \"kubernetes.io/projected/62def2be-9634-4eea-b41a-ae568cd233f8-kube-api-access-rf4hb\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:10:21.105565 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.105529 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ac979f-120f-4676-90a3-130a107f727f-plugins-config-volume" pod "77ac979f-120f-4676-90a3-130a107f727f" (UID: "77ac979f-120f-4676-90a3-130a107f727f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:10:21.107561 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.107540 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ac979f-120f-4676-90a3-130a107f727f-kube-api-access-r88gm" pod "77ac979f-120f-4676-90a3-130a107f727f" (UID: "77ac979f-120f-4676-90a3-130a107f727f"). InnerVolumeSpecName "kube-api-access-r88gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:10:21.206603 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.206562 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r88gm\" (UniqueName: \"kubernetes.io/projected/77ac979f-120f-4676-90a3-130a107f727f-kube-api-access-r88gm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:10:21.206603 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.206602 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ac979f-120f-4676-90a3-130a107f727f-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:10:21.488354 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.488313 5015 generic.go:356] "Generic (PLEG): container finished" podID="62def2be-9634-4eea-b41a-ae568cd233f8" containerID="d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e" exitCode=0 Nov 03 10:10:21.488603 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.488373 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" Nov 03 10:10:21.488603 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.488393 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" event={"ID":"62def2be-9634-4eea-b41a-ae568cd233f8","Type":"ContainerDied","Data":"d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e"} Nov 03 10:10:21.488603 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.488461 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz" event={"ID":"62def2be-9634-4eea-b41a-ae568cd233f8","Type":"ContainerDied","Data":"f18521705e260a51bfd21d7d7b119d535680e7ae79794a550b8f0b8b155992c2"} Nov 03 10:10:21.488603 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.488531 5015 scope.go:122] "RemoveContainer" containerID="d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e" Nov 03 10:10:21.490399 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.490348 5015 generic.go:356] "Generic (PLEG): container finished" podID="77ac979f-120f-4676-90a3-130a107f727f" containerID="72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56" exitCode=0 Nov 03 10:10:21.490762 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.490428 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" event={"ID":"77ac979f-120f-4676-90a3-130a107f727f","Type":"ContainerDied","Data":"72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56"} Nov 03 10:10:21.490762 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.490470 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" event={"ID":"77ac979f-120f-4676-90a3-130a107f727f","Type":"ContainerDied","Data":"cada38c76a359c8c677b13f196aace318336eec50aca58cd0736c57148db10e4"} Nov 03 10:10:21.490762 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.490675 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5976d7f594-zrfzg" Nov 03 10:10:21.499055 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.499038 5015 scope.go:122] "RemoveContainer" containerID="d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e" Nov 03 10:10:21.499313 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:10:21.499297 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e\": container with ID starting with d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e not found: ID does not exist" containerID="d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e" Nov 03 10:10:21.499367 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.499323 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e"} err="failed to get container status \"d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e\": rpc error: code = NotFound desc = could not find container \"d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e\": container with ID starting with d2492e861b4fa140eba7638b5e375f537f060560fda376173e996cec7d8e846e not found: ID does not exist" Nov 03 10:10:21.499367 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.499338 5015 scope.go:122] "RemoveContainer" containerID="72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56" Nov 03 10:10:21.508437 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.508419 5015 scope.go:122] "RemoveContainer" containerID="72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56" Nov 03 10:10:21.508730 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:10:21.508708 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56\": container with ID starting with 72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56 not found: ID does not exist" containerID="72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56" Nov 03 10:10:21.508804 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.508739 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56"} err="failed to get container status \"72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56\": rpc error: code = NotFound desc = could not find container \"72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56\": container with ID starting with 72546bb18ec173834e31f98baa3d443808f454f6de21c3080f78314400628a56 not found: ID does not exist" Nov 03 10:10:21.664236 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.664204 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5976d7f594-zrfzg"] Nov 03 10:10:21.666739 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.666716 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5976d7f594-zrfzg"] Nov 03 10:10:21.776166 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.776075 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz"] Nov 03 10:10:21.779260 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:21.779236 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-6c5d968f79-4qzmz"] Nov 03 10:10:22.638620 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.638587 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6"] Nov 03 10:10:22.639187 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.639168 5015 cpu_manager.go:414] "containerMap: removing container" podUID="62def2be-9634-4eea-b41a-ae568cd233f8" containerName="payload-pre-processing" Nov 03 10:10:22.639187 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.639186 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62def2be-9634-4eea-b41a-ae568cd233f8" containerName="payload-pre-processing" Nov 03 10:10:22.639376 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.639217 5015 cpu_manager.go:414] "containerMap: removing container" podUID="77ac979f-120f-4676-90a3-130a107f727f" containerName="payload-processing" Nov 03 10:10:22.639376 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.639226 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ac979f-120f-4676-90a3-130a107f727f" containerName="payload-processing" Nov 03 10:10:22.639536 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.639377 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="62def2be-9634-4eea-b41a-ae568cd233f8" containerName="payload-pre-processing" Nov 03 10:10:22.639536 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.639394 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ac979f-120f-4676-90a3-130a107f727f" containerName="payload-processing" Nov 03 10:10:22.644469 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.644448 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:22.700957 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.700924 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6"] Nov 03 10:10:22.746651 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.746614 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 03 10:10:22.746861 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.746746 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rhzbt\"" Nov 03 10:10:22.747007 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.746987 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 10:10:22.750378 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.750355 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 10:10:22.750378 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.750355 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 10:10:22.750697 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.750677 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 10:10:22.824639 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.824596 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:22.824639 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.824642 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:22.824933 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.824772 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8lhm\" (UniqueName: \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-kube-api-access-q8lhm\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:22.824933 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.824842 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:22.926016 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.925913 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:22.926316 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.926275 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:22.927041 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.927020 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8lhm\" (UniqueName: \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-kube-api-access-q8lhm\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:22.927290 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.927269 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:22.928450 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.928396 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:22.929150 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.929124 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:22.929150 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:22.929976 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:23.001307 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:23.001267 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8lhm\" (UniqueName: \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-kube-api-access-q8lhm\") pod \"payload-processing-external-model-5bcd7997-bfxm6\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:23.208548 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:23.208439 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62def2be-9634-4eea-b41a-ae568cd233f8" path="/var/lib/kubelet/pods/62def2be-9634-4eea-b41a-ae568cd233f8/volumes" Nov 03 10:10:23.209425 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:23.209370 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ac979f-120f-4676-90a3-130a107f727f" path="/var/lib/kubelet/pods/77ac979f-120f-4676-90a3-130a107f727f/volumes" Nov 03 10:10:23.265977 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:23.265942 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:23.774502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:23.771841 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6"] Nov 03 10:10:24.545290 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:24.545244 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" event={"ID":"ee96933f-764b-48c8-a131-4690dbf64f0e","Type":"ContainerStarted","Data":"4d663b23fa57f20922fe25b5d5b1601da19299e6b999af852561cdbe7f1e5e8e"} Nov 03 10:10:24.546405 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:24.546381 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" event={"ID":"ee96933f-764b-48c8-a131-4690dbf64f0e","Type":"ContainerStarted","Data":"df301aab1f8e30e4a744ec0dd6d99868715aa8e0c7be70539948d80611cefc0d"} Nov 03 10:10:24.547057 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:24.547039 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:29.748214 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:29.748182 5015 scope.go:122] "RemoveContainer" containerID="da10a7d773e3f3ee0eeb6b76b54245ee400e3a740c6c0eff65699ce35c1f629e" Nov 03 10:10:36.560363 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:36.560330 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:10:36.678360 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:10:36.678309 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" podStartSLOduration=14.678295854 podStartE2EDuration="14.678295854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:10:24.678577091 +0000 UTC m=+5457.603519308" watchObservedRunningTime="2026-11-03 10:10:36.678295854 +0000 UTC m=+5469.603238121" Nov 03 10:10:46.901806 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:10:46.901768 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:11:16.901938 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:11:16.901899 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:11:46.901917 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:11:46.901829 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:12:05.274249 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.274214 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf"] Nov 03 10:12:05.278254 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.278233 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:05.297614 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.297587 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-zt9vf\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:05.297770 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.297624 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zws\" (UniqueName: \"kubernetes.io/projected/d0e74267-eff8-4a6e-b04d-0346a5cd233c-kube-api-access-67zws\") pod \"maas-api-revoke-sub-e2547622-zt9vf\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:05.297770 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.297648 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-zt9vf\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:05.335391 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.335360 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf"] Nov 03 10:12:05.381841 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.381810 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:12:05.381841 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.381830 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:12:05.399031 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.399003 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-zt9vf\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:05.399213 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.399175 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-zt9vf\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:05.399288 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.399214 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zws\" (UniqueName: \"kubernetes.io/projected/d0e74267-eff8-4a6e-b04d-0346a5cd233c-kube-api-access-67zws\") pod \"maas-api-revoke-sub-e2547622-zt9vf\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:05.399663 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.399639 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-zt9vf\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:05.401752 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.401721 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-zt9vf\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:05.458389 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.458362 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zws\" (UniqueName: \"kubernetes.io/projected/d0e74267-eff8-4a6e-b04d-0346a5cd233c-kube-api-access-67zws\") pod \"maas-api-revoke-sub-e2547622-zt9vf\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:05.591967 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.591883 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:05.969950 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:05.969926 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf"] Nov 03 10:12:05.971236 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:12:05.971209 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e74267_eff8_4a6e_b04d_0346a5cd233c.slice/crio-4cb7e825179e37ec7aea5d8f2ba3dfa217ae571cefa40ef0352864538e04d7f9 WatchSource:0}: Error finding container 4cb7e825179e37ec7aea5d8f2ba3dfa217ae571cefa40ef0352864538e04d7f9: Status 404 returned error can't find the container with id 4cb7e825179e37ec7aea5d8f2ba3dfa217ae571cefa40ef0352864538e04d7f9 Nov 03 10:12:06.159391 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:06.159356 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" event={"ID":"d0e74267-eff8-4a6e-b04d-0346a5cd233c","Type":"ContainerStarted","Data":"4cb7e825179e37ec7aea5d8f2ba3dfa217ae571cefa40ef0352864538e04d7f9"} Nov 03 10:12:07.166794 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:07.166683 5015 generic.go:356] "Generic (PLEG): container finished" podID="d0e74267-eff8-4a6e-b04d-0346a5cd233c" containerID="eb4bc6d3d2ceccec7bcc771d22d5fa32c4cc83f7a672fb320d2cd4a57f44a4ae" exitCode=0 Nov 03 10:12:07.166794 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:07.166764 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" event={"ID":"d0e74267-eff8-4a6e-b04d-0346a5cd233c","Type":"ContainerDied","Data":"eb4bc6d3d2ceccec7bcc771d22d5fa32c4cc83f7a672fb320d2cd4a57f44a4ae"} Nov 03 10:12:08.320086 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:08.320057 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:08.429298 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:08.429220 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e74267-eff8-4a6e-b04d-0346a5cd233c-kube-api-access-67zws\" (UniqueName: \"kubernetes.io/projected/d0e74267-eff8-4a6e-b04d-0346a5cd233c-kube-api-access-67zws\") pod \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " Nov 03 10:12:08.429468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:08.429325 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-openshift-ca\") pod \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " Nov 03 10:12:08.429468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:08.429375 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-xks-ca\") pod \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\" (UID: \"d0e74267-eff8-4a6e-b04d-0346a5cd233c\") " Nov 03 10:12:08.429713 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:08.429680 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-openshift-ca" pod "d0e74267-eff8-4a6e-b04d-0346a5cd233c" (UID: "d0e74267-eff8-4a6e-b04d-0346a5cd233c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:12:08.431678 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:08.431656 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e74267-eff8-4a6e-b04d-0346a5cd233c-kube-api-access-67zws" pod "d0e74267-eff8-4a6e-b04d-0346a5cd233c" (UID: "d0e74267-eff8-4a6e-b04d-0346a5cd233c"). InnerVolumeSpecName "kube-api-access-67zws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:12:08.431779 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:08.431759 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-xks-ca" pod "d0e74267-eff8-4a6e-b04d-0346a5cd233c" (UID: "d0e74267-eff8-4a6e-b04d-0346a5cd233c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:12:08.530918 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:08.530874 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zws\" (UniqueName: \"kubernetes.io/projected/d0e74267-eff8-4a6e-b04d-0346a5cd233c-kube-api-access-67zws\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:12:08.530918 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:08.530912 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:12:08.530918 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:08.530927 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e74267-eff8-4a6e-b04d-0346a5cd233c-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:12:09.183393 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:09.183355 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" event={"ID":"d0e74267-eff8-4a6e-b04d-0346a5cd233c","Type":"ContainerDied","Data":"4cb7e825179e37ec7aea5d8f2ba3dfa217ae571cefa40ef0352864538e04d7f9"} Nov 03 10:12:09.183393 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:09.183385 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf" Nov 03 10:12:09.183393 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:09.183402 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cb7e825179e37ec7aea5d8f2ba3dfa217ae571cefa40ef0352864538e04d7f9" Nov 03 10:12:16.777470 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:16.777431 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6"] Nov 03 10:12:16.909411 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:12:16.909362 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:12:21.786502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:21.779554 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" podUID="ee96933f-764b-48c8-a131-4690dbf64f0e" containerName="payload-processing" containerID="cri-o://4d663b23fa57f20922fe25b5d5b1601da19299e6b999af852561cdbe7f1e5e8e" gracePeriod=25 Nov 03 10:12:22.277778 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.277746 5015 generic.go:356] "Generic (PLEG): container finished" podID="ee96933f-764b-48c8-a131-4690dbf64f0e" containerID="4d663b23fa57f20922fe25b5d5b1601da19299e6b999af852561cdbe7f1e5e8e" exitCode=0 Nov 03 10:12:22.277948 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.277824 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" event={"ID":"ee96933f-764b-48c8-a131-4690dbf64f0e","Type":"ContainerDied","Data":"4d663b23fa57f20922fe25b5d5b1601da19299e6b999af852561cdbe7f1e5e8e"} Nov 03 10:12:22.277948 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.277891 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" event={"ID":"ee96933f-764b-48c8-a131-4690dbf64f0e","Type":"ContainerDied","Data":"df301aab1f8e30e4a744ec0dd6d99868715aa8e0c7be70539948d80611cefc0d"} Nov 03 10:12:22.277948 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.277915 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df301aab1f8e30e4a744ec0dd6d99868715aa8e0c7be70539948d80611cefc0d" Nov 03 10:12:22.280557 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.280532 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:12:22.424916 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.424826 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-plugins-config-volume\") pod \"ee96933f-764b-48c8-a131-4690dbf64f0e\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " Nov 03 10:12:22.425108 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.424923 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-routing-overlay-volume\") pod \"ee96933f-764b-48c8-a131-4690dbf64f0e\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " Nov 03 10:12:22.425108 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.425019 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-provider-credentials\") pod \"ee96933f-764b-48c8-a131-4690dbf64f0e\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " Nov 03 10:12:22.425108 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.425050 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-kube-api-access-q8lhm\" (UniqueName: \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-kube-api-access-q8lhm\") pod \"ee96933f-764b-48c8-a131-4690dbf64f0e\" (UID: \"ee96933f-764b-48c8-a131-4690dbf64f0e\") " Nov 03 10:12:22.425424 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.425395 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-plugins-config-volume" pod "ee96933f-764b-48c8-a131-4690dbf64f0e" (UID: "ee96933f-764b-48c8-a131-4690dbf64f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:12:22.425604 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.425390 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-routing-overlay-volume" pod "ee96933f-764b-48c8-a131-4690dbf64f0e" (UID: "ee96933f-764b-48c8-a131-4690dbf64f0e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:12:22.427719 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.427687 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-kube-api-access-q8lhm" pod "ee96933f-764b-48c8-a131-4690dbf64f0e" (UID: "ee96933f-764b-48c8-a131-4690dbf64f0e"). InnerVolumeSpecName "kube-api-access-q8lhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:12:22.427867 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.427850 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-provider-credentials" pod "ee96933f-764b-48c8-a131-4690dbf64f0e" (UID: "ee96933f-764b-48c8-a131-4690dbf64f0e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:12:22.526328 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.526282 5015 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-provider-credentials\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:12:22.526328 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.526322 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8lhm\" (UniqueName: \"kubernetes.io/projected/ee96933f-764b-48c8-a131-4690dbf64f0e-kube-api-access-q8lhm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:12:22.526568 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.526337 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:12:22.526568 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:22.526353 5015 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee96933f-764b-48c8-a131-4690dbf64f0e-routing-overlay-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:12:23.283260 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:23.283227 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6" Nov 03 10:12:23.456301 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:23.456263 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6"] Nov 03 10:12:23.506791 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:23.506751 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bfxm6"] Nov 03 10:12:25.204620 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:25.204583 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee96933f-764b-48c8-a131-4690dbf64f0e" path="/var/lib/kubelet/pods/ee96933f-764b-48c8-a131-4690dbf64f0e/volumes" Nov 03 10:12:29.790265 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:29.790230 5015 scope.go:122] "RemoveContainer" containerID="b0d3fac74525822d0dc687b86a8f2d4a2311057479b50c44bf9fd54cd8576a3f" Nov 03 10:12:36.186314 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.186278 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz"] Nov 03 10:12:36.186832 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.186806 5015 cpu_manager.go:414] "containerMap: removing container" podUID="ee96933f-764b-48c8-a131-4690dbf64f0e" containerName="payload-processing" Nov 03 10:12:36.186832 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.186819 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee96933f-764b-48c8-a131-4690dbf64f0e" containerName="payload-processing" Nov 03 10:12:36.186912 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.186834 5015 cpu_manager.go:414] "containerMap: removing container" podUID="d0e74267-eff8-4a6e-b04d-0346a5cd233c" containerName="revoke-keys" Nov 03 10:12:36.186912 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.186840 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e74267-eff8-4a6e-b04d-0346a5cd233c" containerName="revoke-keys" Nov 03 10:12:36.186980 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.186925 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee96933f-764b-48c8-a131-4690dbf64f0e" containerName="payload-processing" Nov 03 10:12:36.186980 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.186938 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e74267-eff8-4a6e-b04d-0346a5cd233c" containerName="revoke-keys" Nov 03 10:12:36.191696 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.191672 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:36.294502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.294431 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7nwfz\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:36.294728 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.294593 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktb5m\" (UniqueName: \"kubernetes.io/projected/319d5b19-0780-43a1-8980-42723cb1c38c-kube-api-access-ktb5m\") pod \"maas-api-revoke-sub-c1e271a9-7nwfz\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:36.294728 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.294636 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7nwfz\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:36.296830 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.296804 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:12:36.296830 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.296829 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:12:36.299574 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.299552 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz"] Nov 03 10:12:36.395523 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.395455 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktb5m\" (UniqueName: \"kubernetes.io/projected/319d5b19-0780-43a1-8980-42723cb1c38c-kube-api-access-ktb5m\") pod \"maas-api-revoke-sub-c1e271a9-7nwfz\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:36.395523 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.395525 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7nwfz\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:36.395747 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.395623 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7nwfz\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:36.396144 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.396122 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7nwfz\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:36.398264 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.398241 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7nwfz\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:36.455230 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.455150 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktb5m\" (UniqueName: \"kubernetes.io/projected/319d5b19-0780-43a1-8980-42723cb1c38c-kube-api-access-ktb5m\") pod \"maas-api-revoke-sub-c1e271a9-7nwfz\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:36.509517 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.509445 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:36.887894 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:36.887863 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz"] Nov 03 10:12:36.889216 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:12:36.889184 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319d5b19_0780_43a1_8980_42723cb1c38c.slice/crio-c2914dcb6eece32bb19041d1cad34e251bd951aab62397b1f1c5b3bf2bce46a9 WatchSource:0}: Error finding container c2914dcb6eece32bb19041d1cad34e251bd951aab62397b1f1c5b3bf2bce46a9: Status 404 returned error can't find the container with id c2914dcb6eece32bb19041d1cad34e251bd951aab62397b1f1c5b3bf2bce46a9 Nov 03 10:12:37.413238 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:37.413151 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" event={"ID":"319d5b19-0780-43a1-8980-42723cb1c38c","Type":"ContainerStarted","Data":"58303066e93f2bcd5bd0a3708e6aa182a7f1f5c9f56a8c8977a04cbac893e036"} Nov 03 10:12:37.413238 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:37.413210 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" event={"ID":"319d5b19-0780-43a1-8980-42723cb1c38c","Type":"ContainerStarted","Data":"c2914dcb6eece32bb19041d1cad34e251bd951aab62397b1f1c5b3bf2bce46a9"} Nov 03 10:12:37.534160 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:37.534105 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" podStartSLOduration=1.534089839 podStartE2EDuration="1.534089839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:12:37.533390421 +0000 UTC m=+5590.458332644" watchObservedRunningTime="2026-11-03 10:12:37.534089839 +0000 UTC m=+5590.459032056" Nov 03 10:12:38.420537 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:38.420501 5015 generic.go:356] "Generic (PLEG): container finished" podID="319d5b19-0780-43a1-8980-42723cb1c38c" containerID="58303066e93f2bcd5bd0a3708e6aa182a7f1f5c9f56a8c8977a04cbac893e036" exitCode=0 Nov 03 10:12:38.420991 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:38.420568 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" event={"ID":"319d5b19-0780-43a1-8980-42723cb1c38c","Type":"ContainerDied","Data":"58303066e93f2bcd5bd0a3708e6aa182a7f1f5c9f56a8c8977a04cbac893e036"} Nov 03 10:12:39.568047 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:39.568023 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:39.626756 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:39.626730 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-xks-ca\") pod \"319d5b19-0780-43a1-8980-42723cb1c38c\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " Nov 03 10:12:39.626936 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:39.626814 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-openshift-ca\") pod \"319d5b19-0780-43a1-8980-42723cb1c38c\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " Nov 03 10:12:39.626936 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:39.626867 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319d5b19-0780-43a1-8980-42723cb1c38c-kube-api-access-ktb5m\" (UniqueName: \"kubernetes.io/projected/319d5b19-0780-43a1-8980-42723cb1c38c-kube-api-access-ktb5m\") pod \"319d5b19-0780-43a1-8980-42723cb1c38c\" (UID: \"319d5b19-0780-43a1-8980-42723cb1c38c\") " Nov 03 10:12:39.627149 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:39.627128 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-openshift-ca" pod "319d5b19-0780-43a1-8980-42723cb1c38c" (UID: "319d5b19-0780-43a1-8980-42723cb1c38c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:12:39.629150 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:39.629127 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-xks-ca" pod "319d5b19-0780-43a1-8980-42723cb1c38c" (UID: "319d5b19-0780-43a1-8980-42723cb1c38c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:12:39.629230 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:39.629218 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319d5b19-0780-43a1-8980-42723cb1c38c-kube-api-access-ktb5m" pod "319d5b19-0780-43a1-8980-42723cb1c38c" (UID: "319d5b19-0780-43a1-8980-42723cb1c38c"). InnerVolumeSpecName "kube-api-access-ktb5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:12:39.727648 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:39.727618 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktb5m\" (UniqueName: \"kubernetes.io/projected/319d5b19-0780-43a1-8980-42723cb1c38c-kube-api-access-ktb5m\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:12:39.727648 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:39.727647 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:12:39.727848 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:39.727658 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319d5b19-0780-43a1-8980-42723cb1c38c-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:12:40.434120 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:40.434086 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" Nov 03 10:12:40.434306 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:40.434089 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz" event={"ID":"319d5b19-0780-43a1-8980-42723cb1c38c","Type":"ContainerDied","Data":"c2914dcb6eece32bb19041d1cad34e251bd951aab62397b1f1c5b3bf2bce46a9"} Nov 03 10:12:40.434306 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:12:40.434217 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2914dcb6eece32bb19041d1cad34e251bd951aab62397b1f1c5b3bf2bce46a9" Nov 03 10:12:46.902857 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:12:46.902820 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:13:01.619391 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.619340 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86"] Nov 03 10:13:01.620033 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.620011 5015 cpu_manager.go:414] "containerMap: removing container" podUID="319d5b19-0780-43a1-8980-42723cb1c38c" containerName="revoke-keys" Nov 03 10:13:01.620131 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.620036 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319d5b19-0780-43a1-8980-42723cb1c38c" containerName="revoke-keys" Nov 03 10:13:01.620194 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.620150 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="319d5b19-0780-43a1-8980-42723cb1c38c" containerName="revoke-keys" Nov 03 10:13:01.623693 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.623666 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:01.670880 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.670850 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmp86\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:01.671101 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.670896 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzpsf\" (UniqueName: \"kubernetes.io/projected/457fe3eb-abe4-4c84-b733-8977b9651b33-kube-api-access-hzpsf\") pod \"maas-api-revoke-sub-2e1f5b5f-jmp86\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:01.671248 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.671097 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmp86\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:01.680715 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.680682 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86"] Nov 03 10:13:01.726057 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.726029 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:13:01.726238 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.726105 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:13:01.771809 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.771770 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmp86\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:01.772000 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.771832 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmp86\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:01.772000 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.771861 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpsf\" (UniqueName: \"kubernetes.io/projected/457fe3eb-abe4-4c84-b733-8977b9651b33-kube-api-access-hzpsf\") pod \"maas-api-revoke-sub-2e1f5b5f-jmp86\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:01.772419 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.772391 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmp86\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:01.774394 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.774364 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmp86\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:01.831491 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.831431 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzpsf\" (UniqueName: \"kubernetes.io/projected/457fe3eb-abe4-4c84-b733-8977b9651b33-kube-api-access-hzpsf\") pod \"maas-api-revoke-sub-2e1f5b5f-jmp86\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:01.937401 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:01.937322 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:02.316860 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:02.316827 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86"] Nov 03 10:13:02.317009 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:13:02.316979 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457fe3eb_abe4_4c84_b733_8977b9651b33.slice/crio-97480965dcb40a18cdc83ca6cb705a50d0e3f96d4c497b790b296c8b91788772 WatchSource:0}: Error finding container 97480965dcb40a18cdc83ca6cb705a50d0e3f96d4c497b790b296c8b91788772: Status 404 returned error can't find the container with id 97480965dcb40a18cdc83ca6cb705a50d0e3f96d4c497b790b296c8b91788772 Nov 03 10:13:02.572581 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:02.572507 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" event={"ID":"457fe3eb-abe4-4c84-b733-8977b9651b33","Type":"ContainerStarted","Data":"97480965dcb40a18cdc83ca6cb705a50d0e3f96d4c497b790b296c8b91788772"} Nov 03 10:13:03.580225 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:03.580134 5015 generic.go:356] "Generic (PLEG): container finished" podID="457fe3eb-abe4-4c84-b733-8977b9651b33" containerID="3abae6dac360af050208a9e09d5ac208907d7147b72b729d68d960b76118080c" exitCode=0 Nov 03 10:13:03.580646 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:03.580223 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" event={"ID":"457fe3eb-abe4-4c84-b733-8977b9651b33","Type":"ContainerDied","Data":"3abae6dac360af050208a9e09d5ac208907d7147b72b729d68d960b76118080c"} Nov 03 10:13:04.723563 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:04.723536 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:04.798332 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:04.798302 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-openshift-ca\") pod \"457fe3eb-abe4-4c84-b733-8977b9651b33\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " Nov 03 10:13:04.798531 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:04.798412 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-xks-ca\") pod \"457fe3eb-abe4-4c84-b733-8977b9651b33\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " Nov 03 10:13:04.798531 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:04.798466 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457fe3eb-abe4-4c84-b733-8977b9651b33-kube-api-access-hzpsf\" (UniqueName: \"kubernetes.io/projected/457fe3eb-abe4-4c84-b733-8977b9651b33-kube-api-access-hzpsf\") pod \"457fe3eb-abe4-4c84-b733-8977b9651b33\" (UID: \"457fe3eb-abe4-4c84-b733-8977b9651b33\") " Nov 03 10:13:04.798703 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:04.798680 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-openshift-ca" pod "457fe3eb-abe4-4c84-b733-8977b9651b33" (UID: "457fe3eb-abe4-4c84-b733-8977b9651b33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:13:04.800838 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:04.800811 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-xks-ca" pod "457fe3eb-abe4-4c84-b733-8977b9651b33" (UID: "457fe3eb-abe4-4c84-b733-8977b9651b33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:13:04.800838 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:04.800819 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457fe3eb-abe4-4c84-b733-8977b9651b33-kube-api-access-hzpsf" pod "457fe3eb-abe4-4c84-b733-8977b9651b33" (UID: "457fe3eb-abe4-4c84-b733-8977b9651b33"). InnerVolumeSpecName "kube-api-access-hzpsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:13:04.899973 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:04.899886 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:13:04.899973 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:04.899919 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457fe3eb-abe4-4c84-b733-8977b9651b33-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:13:04.899973 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:04.899930 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzpsf\" (UniqueName: \"kubernetes.io/projected/457fe3eb-abe4-4c84-b733-8977b9651b33-kube-api-access-hzpsf\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:13:05.592972 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:05.592940 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" Nov 03 10:13:05.592972 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:05.592964 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86" event={"ID":"457fe3eb-abe4-4c84-b733-8977b9651b33","Type":"ContainerDied","Data":"97480965dcb40a18cdc83ca6cb705a50d0e3f96d4c497b790b296c8b91788772"} Nov 03 10:13:05.593195 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:05.593007 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97480965dcb40a18cdc83ca6cb705a50d0e3f96d4c497b790b296c8b91788772" Nov 03 10:13:12.054762 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:12.054725 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc"] Nov 03 10:13:12.105111 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:12.105073 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2rrqc"] Nov 03 10:13:13.203239 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:13.203203 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="411954b6-8724-4754-afde-8cc9c2008d4f" path="/var/lib/kubelet/pods/411954b6-8724-4754-afde-8cc9c2008d4f/volumes" Nov 03 10:13:16.902182 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:13:16.902145 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:13:24.150674 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.150636 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5"] Nov 03 10:13:24.151107 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.151094 5015 cpu_manager.go:414] "containerMap: removing container" podUID="457fe3eb-abe4-4c84-b733-8977b9651b33" containerName="revoke-keys" Nov 03 10:13:24.151154 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.151108 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457fe3eb-abe4-4c84-b733-8977b9651b33" containerName="revoke-keys" Nov 03 10:13:24.151196 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.151179 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="457fe3eb-abe4-4c84-b733-8977b9651b33" containerName="revoke-keys" Nov 03 10:13:24.154110 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.154092 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:24.209045 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.208987 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:13:24.209229 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.209191 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:13:24.212449 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.212426 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5"] Nov 03 10:13:24.293089 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.293059 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jtgs5\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:24.293089 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.293093 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfgqb\" (UniqueName: \"kubernetes.io/projected/9dc66bb5-36ba-488e-8f4d-0dc347da4675-kube-api-access-gfgqb\") pod \"maas-api-revoke-sub-ec843018-jtgs5\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:24.293322 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.293133 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jtgs5\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:24.394038 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.393999 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jtgs5\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:24.394038 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.394045 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgqb\" (UniqueName: \"kubernetes.io/projected/9dc66bb5-36ba-488e-8f4d-0dc347da4675-kube-api-access-gfgqb\") pod \"maas-api-revoke-sub-ec843018-jtgs5\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:24.394333 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.394078 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jtgs5\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:24.394805 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.394776 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jtgs5\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:24.396686 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.396667 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jtgs5\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:24.453636 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.453572 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfgqb\" (UniqueName: \"kubernetes.io/projected/9dc66bb5-36ba-488e-8f4d-0dc347da4675-kube-api-access-gfgqb\") pod \"maas-api-revoke-sub-ec843018-jtgs5\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:24.467415 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.467388 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:24.845072 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:24.845041 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5"] Nov 03 10:13:24.846542 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:13:24.846512 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc66bb5_36ba_488e_8f4d_0dc347da4675.slice/crio-75214f452f3d360f95a19b03c56ff0d4970f516a2cf10ac40633dc143182d532 WatchSource:0}: Error finding container 75214f452f3d360f95a19b03c56ff0d4970f516a2cf10ac40633dc143182d532: Status 404 returned error can't find the container with id 75214f452f3d360f95a19b03c56ff0d4970f516a2cf10ac40633dc143182d532 Nov 03 10:13:25.713496 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:25.713443 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" event={"ID":"9dc66bb5-36ba-488e-8f4d-0dc347da4675","Type":"ContainerStarted","Data":"0d029d4614a22297e48f0976230e7315a9b625b8e34ed04ec573d46d4c0d1817"} Nov 03 10:13:25.713881 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:25.713531 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" event={"ID":"9dc66bb5-36ba-488e-8f4d-0dc347da4675","Type":"ContainerStarted","Data":"75214f452f3d360f95a19b03c56ff0d4970f516a2cf10ac40633dc143182d532"} Nov 03 10:13:25.830619 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:25.830563 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" podStartSLOduration=1.830545663 podStartE2EDuration="1.830545663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:13:25.829379144 +0000 UTC m=+5638.754321478" watchObservedRunningTime="2026-11-03 10:13:25.830545663 +0000 UTC m=+5638.755487881" Nov 03 10:13:26.722611 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:26.722570 5015 generic.go:356] "Generic (PLEG): container finished" podID="9dc66bb5-36ba-488e-8f4d-0dc347da4675" containerID="0d029d4614a22297e48f0976230e7315a9b625b8e34ed04ec573d46d4c0d1817" exitCode=0 Nov 03 10:13:26.723044 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:26.722648 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" event={"ID":"9dc66bb5-36ba-488e-8f4d-0dc347da4675","Type":"ContainerDied","Data":"0d029d4614a22297e48f0976230e7315a9b625b8e34ed04ec573d46d4c0d1817"} Nov 03 10:13:27.892148 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:27.892116 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:28.034928 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.034827 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc66bb5-36ba-488e-8f4d-0dc347da4675-kube-api-access-gfgqb\" (UniqueName: \"kubernetes.io/projected/9dc66bb5-36ba-488e-8f4d-0dc347da4675-kube-api-access-gfgqb\") pod \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " Nov 03 10:13:28.034928 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.034906 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-openshift-ca\") pod \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " Nov 03 10:13:28.035566 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.035031 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-xks-ca\") pod \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\" (UID: \"9dc66bb5-36ba-488e-8f4d-0dc347da4675\") " Nov 03 10:13:28.035566 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.035263 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-openshift-ca" pod "9dc66bb5-36ba-488e-8f4d-0dc347da4675" (UID: "9dc66bb5-36ba-488e-8f4d-0dc347da4675"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:13:28.035566 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.035421 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:13:28.037252 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.037229 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-xks-ca" pod "9dc66bb5-36ba-488e-8f4d-0dc347da4675" (UID: "9dc66bb5-36ba-488e-8f4d-0dc347da4675"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:13:28.037380 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.037361 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc66bb5-36ba-488e-8f4d-0dc347da4675-kube-api-access-gfgqb" pod "9dc66bb5-36ba-488e-8f4d-0dc347da4675" (UID: "9dc66bb5-36ba-488e-8f4d-0dc347da4675"). InnerVolumeSpecName "kube-api-access-gfgqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:13:28.136725 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.136688 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc66bb5-36ba-488e-8f4d-0dc347da4675-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:13:28.136725 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.136723 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfgqb\" (UniqueName: \"kubernetes.io/projected/9dc66bb5-36ba-488e-8f4d-0dc347da4675-kube-api-access-gfgqb\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:13:28.737553 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.737524 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" Nov 03 10:13:28.737553 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.737524 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5" event={"ID":"9dc66bb5-36ba-488e-8f4d-0dc347da4675","Type":"ContainerDied","Data":"75214f452f3d360f95a19b03c56ff0d4970f516a2cf10ac40633dc143182d532"} Nov 03 10:13:28.737747 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:28.737580 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75214f452f3d360f95a19b03c56ff0d4970f516a2cf10ac40633dc143182d532" Nov 03 10:13:46.054331 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:46.054290 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9"] Nov 03 10:13:46.106510 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:46.106448 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-x74z9"] Nov 03 10:13:46.902243 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:13:46.902207 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:13:47.203411 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:13:47.203333 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739c6d98-468c-4a34-9820-0ca7a7efdc8c" path="/var/lib/kubelet/pods/739c6d98-468c-4a34-9820-0ca7a7efdc8c/volumes" Nov 03 10:14:09.281378 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.281341 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww"] Nov 03 10:14:09.281873 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.281859 5015 cpu_manager.go:414] "containerMap: removing container" podUID="9dc66bb5-36ba-488e-8f4d-0dc347da4675" containerName="revoke-keys" Nov 03 10:14:09.281920 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.281877 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc66bb5-36ba-488e-8f4d-0dc347da4675" containerName="revoke-keys" Nov 03 10:14:09.281960 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.281949 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc66bb5-36ba-488e-8f4d-0dc347da4675" containerName="revoke-keys" Nov 03 10:14:09.285459 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.285444 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:09.342210 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.342181 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww"] Nov 03 10:14:09.371612 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.371579 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfl4\" (UniqueName: \"kubernetes.io/projected/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-kube-api-access-2hfl4\") pod \"maas-api-revoke-sub-046e1343-wscww\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:09.371782 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.371625 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wscww\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:09.371782 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.371693 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wscww\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:09.389225 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.389197 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:14:09.389225 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.389216 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:14:09.472592 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.472563 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wscww\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:09.472776 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.472652 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfl4\" (UniqueName: \"kubernetes.io/projected/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-kube-api-access-2hfl4\") pod \"maas-api-revoke-sub-046e1343-wscww\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:09.472776 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.472691 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wscww\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:09.473325 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.473298 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wscww\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:09.475149 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.475124 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wscww\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:09.531848 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.531787 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfl4\" (UniqueName: \"kubernetes.io/projected/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-kube-api-access-2hfl4\") pod \"maas-api-revoke-sub-046e1343-wscww\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:09.597885 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.597859 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:09.977804 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:09.977775 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww"] Nov 03 10:14:09.979803 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:14:09.979775 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5c1f3a_07da_4d3c_8428_e06f6647b23a.slice/crio-61045f0e3d76db0248f7bd2154694548d4f8393fba8fa0fa8bf36022952fd81a WatchSource:0}: Error finding container 61045f0e3d76db0248f7bd2154694548d4f8393fba8fa0fa8bf36022952fd81a: Status 404 returned error can't find the container with id 61045f0e3d76db0248f7bd2154694548d4f8393fba8fa0fa8bf36022952fd81a Nov 03 10:14:10.004781 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:10.004754 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" event={"ID":"0a5c1f3a-07da-4d3c-8428-e06f6647b23a","Type":"ContainerStarted","Data":"61045f0e3d76db0248f7bd2154694548d4f8393fba8fa0fa8bf36022952fd81a"} Nov 03 10:14:11.013102 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:11.013062 5015 generic.go:356] "Generic (PLEG): container finished" podID="0a5c1f3a-07da-4d3c-8428-e06f6647b23a" containerID="a5702acd79067d787af5e7b994860423e45046a633d566934e80b57ad2b20639" exitCode=0 Nov 03 10:14:11.013617 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:11.013102 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" event={"ID":"0a5c1f3a-07da-4d3c-8428-e06f6647b23a","Type":"ContainerDied","Data":"a5702acd79067d787af5e7b994860423e45046a633d566934e80b57ad2b20639"} Nov 03 10:14:11.054058 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:11.054029 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq"] Nov 03 10:14:11.104654 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:11.104583 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d42e7a-nwfsq"] Nov 03 10:14:11.203977 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:11.203942 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3f113a-bc81-4013-aca8-cd6d474b6a9c" path="/var/lib/kubelet/pods/2c3f113a-bc81-4013-aca8-cd6d474b6a9c/volumes" Nov 03 10:14:12.161310 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:12.161285 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:12.301143 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:12.301060 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-kube-api-access-2hfl4\" (UniqueName: \"kubernetes.io/projected/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-kube-api-access-2hfl4\") pod \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " Nov 03 10:14:12.301306 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:12.301173 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-openshift-ca\") pod \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " Nov 03 10:14:12.301306 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:12.301228 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-xks-ca\") pod \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\" (UID: \"0a5c1f3a-07da-4d3c-8428-e06f6647b23a\") " Nov 03 10:14:12.301564 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:12.301527 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-openshift-ca" pod "0a5c1f3a-07da-4d3c-8428-e06f6647b23a" (UID: "0a5c1f3a-07da-4d3c-8428-e06f6647b23a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:14:12.301806 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:12.301778 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:14:12.303694 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:12.303665 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-kube-api-access-2hfl4" pod "0a5c1f3a-07da-4d3c-8428-e06f6647b23a" (UID: "0a5c1f3a-07da-4d3c-8428-e06f6647b23a"). InnerVolumeSpecName "kube-api-access-2hfl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:14:12.303795 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:12.303727 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-xks-ca" pod "0a5c1f3a-07da-4d3c-8428-e06f6647b23a" (UID: "0a5c1f3a-07da-4d3c-8428-e06f6647b23a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:14:12.402615 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:12.402579 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hfl4\" (UniqueName: \"kubernetes.io/projected/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-kube-api-access-2hfl4\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:14:12.402615 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:12.402610 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c1f3a-07da-4d3c-8428-e06f6647b23a-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:14:13.029377 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:13.029351 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" Nov 03 10:14:13.029578 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:13.029391 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww" event={"ID":"0a5c1f3a-07da-4d3c-8428-e06f6647b23a","Type":"ContainerDied","Data":"61045f0e3d76db0248f7bd2154694548d4f8393fba8fa0fa8bf36022952fd81a"} Nov 03 10:14:13.029578 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:13.029440 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61045f0e3d76db0248f7bd2154694548d4f8393fba8fa0fa8bf36022952fd81a" Nov 03 10:14:16.901412 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:14:16.901382 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:14:20.077902 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.077818 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq"] Nov 03 10:14:20.078523 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.078507 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0a5c1f3a-07da-4d3c-8428-e06f6647b23a" containerName="revoke-keys" Nov 03 10:14:20.078583 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.078526 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5c1f3a-07da-4d3c-8428-e06f6647b23a" containerName="revoke-keys" Nov 03 10:14:20.078653 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.078641 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5c1f3a-07da-4d3c-8428-e06f6647b23a" containerName="revoke-keys" Nov 03 10:14:20.082327 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.082298 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:20.137952 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.137918 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq"] Nov 03 10:14:20.179013 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.178975 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-s2fgq\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:20.179181 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.179145 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvhtj\" (UniqueName: \"kubernetes.io/projected/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-kube-api-access-cvhtj\") pod \"maas-api-revoke-sub-d3fba056-s2fgq\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:20.179241 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.179201 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-s2fgq\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:20.184407 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.184387 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:14:20.184528 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.184520 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:14:20.279812 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.279778 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvhtj\" (UniqueName: \"kubernetes.io/projected/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-kube-api-access-cvhtj\") pod \"maas-api-revoke-sub-d3fba056-s2fgq\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:20.279974 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.279841 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-s2fgq\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:20.279974 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.279884 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-s2fgq\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:20.280608 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.280589 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-s2fgq\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:20.282692 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.282665 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-s2fgq\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:20.339318 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.339246 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvhtj\" (UniqueName: \"kubernetes.io/projected/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-kube-api-access-cvhtj\") pod \"maas-api-revoke-sub-d3fba056-s2fgq\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:20.398522 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.398463 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:20.776553 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:20.776522 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq"] Nov 03 10:14:20.778214 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:14:20.778187 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2cd64a_ff55_48d1_9372_6cc05c6d43dd.slice/crio-89aa34df20f774fe0787d1b58a6e5254a9e9815f3d918132b4fc8e3e9eaca8de WatchSource:0}: Error finding container 89aa34df20f774fe0787d1b58a6e5254a9e9815f3d918132b4fc8e3e9eaca8de: Status 404 returned error can't find the container with id 89aa34df20f774fe0787d1b58a6e5254a9e9815f3d918132b4fc8e3e9eaca8de Nov 03 10:14:21.081893 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:21.081855 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" event={"ID":"be2cd64a-ff55-48d1-9372-6cc05c6d43dd","Type":"ContainerStarted","Data":"d9351fc0ebe7a75d909923d6fd6d60960735228b3d4dc5787e2f55115ef0d3eb"} Nov 03 10:14:21.082280 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:21.081934 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" event={"ID":"be2cd64a-ff55-48d1-9372-6cc05c6d43dd","Type":"ContainerStarted","Data":"89aa34df20f774fe0787d1b58a6e5254a9e9815f3d918132b4fc8e3e9eaca8de"} Nov 03 10:14:21.199943 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:21.199889 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" podStartSLOduration=1.199874527 podStartE2EDuration="1.199874527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:14:21.197793507 +0000 UTC m=+5694.122735725" watchObservedRunningTime="2026-11-03 10:14:21.199874527 +0000 UTC m=+5694.124816744" Nov 03 10:14:22.089196 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:22.089161 5015 generic.go:356] "Generic (PLEG): container finished" podID="be2cd64a-ff55-48d1-9372-6cc05c6d43dd" containerID="d9351fc0ebe7a75d909923d6fd6d60960735228b3d4dc5787e2f55115ef0d3eb" exitCode=0 Nov 03 10:14:22.089634 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:22.089255 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" event={"ID":"be2cd64a-ff55-48d1-9372-6cc05c6d43dd","Type":"ContainerDied","Data":"d9351fc0ebe7a75d909923d6fd6d60960735228b3d4dc5787e2f55115ef0d3eb"} Nov 03 10:14:23.283651 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:23.283621 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:23.413746 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:23.413644 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-openshift-ca\") pod \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " Nov 03 10:14:23.413925 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:23.413752 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-xks-ca\") pod \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " Nov 03 10:14:23.413925 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:23.413793 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-kube-api-access-cvhtj\" (UniqueName: \"kubernetes.io/projected/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-kube-api-access-cvhtj\") pod \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\" (UID: \"be2cd64a-ff55-48d1-9372-6cc05c6d43dd\") " Nov 03 10:14:23.414816 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:23.414787 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-openshift-ca" pod "be2cd64a-ff55-48d1-9372-6cc05c6d43dd" (UID: "be2cd64a-ff55-48d1-9372-6cc05c6d43dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:14:23.424446 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:23.424415 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-xks-ca" pod "be2cd64a-ff55-48d1-9372-6cc05c6d43dd" (UID: "be2cd64a-ff55-48d1-9372-6cc05c6d43dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:14:23.424610 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:23.424495 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-kube-api-access-cvhtj" pod "be2cd64a-ff55-48d1-9372-6cc05c6d43dd" (UID: "be2cd64a-ff55-48d1-9372-6cc05c6d43dd"). InnerVolumeSpecName "kube-api-access-cvhtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:14:23.516233 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:23.516132 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:14:23.516233 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:23.516174 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvhtj\" (UniqueName: \"kubernetes.io/projected/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-kube-api-access-cvhtj\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:14:23.516233 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:23.516193 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2cd64a-ff55-48d1-9372-6cc05c6d43dd-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:14:24.119291 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:24.119230 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" event={"ID":"be2cd64a-ff55-48d1-9372-6cc05c6d43dd","Type":"ContainerDied","Data":"89aa34df20f774fe0787d1b58a6e5254a9e9815f3d918132b4fc8e3e9eaca8de"} Nov 03 10:14:24.119516 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:24.119314 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89aa34df20f774fe0787d1b58a6e5254a9e9815f3d918132b4fc8e3e9eaca8de" Nov 03 10:14:24.119516 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:24.119417 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq" Nov 03 10:14:28.267452 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:28.267417 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:14:28.291502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:28.291465 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:14:29.824352 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:29.824322 5015 scope.go:122] "RemoveContainer" containerID="d5b1c21a6114f6325022cc5912bf30ebdf98a0071eab6a987c321d7ae7813492" Nov 03 10:14:46.902067 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:14:46.901988 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:14:57.949170 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:57.949133 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm"] Nov 03 10:14:57.949803 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:57.949787 5015 cpu_manager.go:414] "containerMap: removing container" podUID="be2cd64a-ff55-48d1-9372-6cc05c6d43dd" containerName="revoke-keys" Nov 03 10:14:57.949855 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:57.949806 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2cd64a-ff55-48d1-9372-6cc05c6d43dd" containerName="revoke-keys" Nov 03 10:14:57.950315 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:57.949929 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2cd64a-ff55-48d1-9372-6cc05c6d43dd" containerName="revoke-keys" Nov 03 10:14:57.953621 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:57.953602 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:14:58.007789 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.007753 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm"] Nov 03 10:14:58.054409 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.054374 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:14:58.054615 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.054453 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:14:58.085541 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.085511 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-pdxlm\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:14:58.085714 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.085677 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-pdxlm\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:14:58.085795 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.085723 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8m7p\" (UniqueName: \"kubernetes.io/projected/6b3ee15d-4073-44d1-b4db-2eb3edea6566-kube-api-access-q8m7p\") pod \"maas-api-revoke-sub-ae594979-pdxlm\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:14:58.187311 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.187282 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-pdxlm\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:14:58.187311 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.187320 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8m7p\" (UniqueName: \"kubernetes.io/projected/6b3ee15d-4073-44d1-b4db-2eb3edea6566-kube-api-access-q8m7p\") pod \"maas-api-revoke-sub-ae594979-pdxlm\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:14:58.187609 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.187370 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-pdxlm\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:14:58.188065 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.188037 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-pdxlm\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:14:58.189995 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.189970 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-pdxlm\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:14:58.246683 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.246655 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8m7p\" (UniqueName: \"kubernetes.io/projected/6b3ee15d-4073-44d1-b4db-2eb3edea6566-kube-api-access-q8m7p\") pod \"maas-api-revoke-sub-ae594979-pdxlm\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:14:58.266621 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.266597 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:14:58.643034 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:58.643005 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm"] Nov 03 10:14:58.643247 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:14:58.643223 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3ee15d_4073_44d1_b4db_2eb3edea6566.slice/crio-e097753dbd4ed48f6b5210893c82932bd789e8fbcf9846fc5df93a1429283f9c WatchSource:0}: Error finding container e097753dbd4ed48f6b5210893c82932bd789e8fbcf9846fc5df93a1429283f9c: Status 404 returned error can't find the container with id e097753dbd4ed48f6b5210893c82932bd789e8fbcf9846fc5df93a1429283f9c Nov 03 10:14:59.390574 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:59.390538 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" event={"ID":"6b3ee15d-4073-44d1-b4db-2eb3edea6566","Type":"ContainerStarted","Data":"77a72afcd324f1884d7f1f47db79848e1cc52f04673bcda8dafc0220d00f7fb8"} Nov 03 10:14:59.390970 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:59.390589 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" event={"ID":"6b3ee15d-4073-44d1-b4db-2eb3edea6566","Type":"ContainerStarted","Data":"e097753dbd4ed48f6b5210893c82932bd789e8fbcf9846fc5df93a1429283f9c"} Nov 03 10:14:59.508735 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:14:59.508682 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" podStartSLOduration=2.508669025 podStartE2EDuration="2.508669025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:14:59.506186719 +0000 UTC m=+5732.431128937" watchObservedRunningTime="2026-11-03 10:14:59.508669025 +0000 UTC m=+5732.433611259" Nov 03 10:15:00.398858 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:00.398819 5015 generic.go:356] "Generic (PLEG): container finished" podID="6b3ee15d-4073-44d1-b4db-2eb3edea6566" containerID="77a72afcd324f1884d7f1f47db79848e1cc52f04673bcda8dafc0220d00f7fb8" exitCode=0 Nov 03 10:15:00.399304 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:00.398906 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" event={"ID":"6b3ee15d-4073-44d1-b4db-2eb3edea6566","Type":"ContainerDied","Data":"77a72afcd324f1884d7f1f47db79848e1cc52f04673bcda8dafc0220d00f7fb8"} Nov 03 10:15:01.545328 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:01.545303 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:15:01.621047 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:01.621019 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3ee15d-4073-44d1-b4db-2eb3edea6566-kube-api-access-q8m7p\" (UniqueName: \"kubernetes.io/projected/6b3ee15d-4073-44d1-b4db-2eb3edea6566-kube-api-access-q8m7p\") pod \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " Nov 03 10:15:01.621232 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:01.621128 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-xks-ca\") pod \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " Nov 03 10:15:01.621232 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:01.621159 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-openshift-ca\") pod \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\" (UID: \"6b3ee15d-4073-44d1-b4db-2eb3edea6566\") " Nov 03 10:15:01.621619 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:01.621590 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-openshift-ca" pod "6b3ee15d-4073-44d1-b4db-2eb3edea6566" (UID: "6b3ee15d-4073-44d1-b4db-2eb3edea6566"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:15:01.623379 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:01.623352 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-xks-ca" pod "6b3ee15d-4073-44d1-b4db-2eb3edea6566" (UID: "6b3ee15d-4073-44d1-b4db-2eb3edea6566"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:15:01.623512 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:01.623394 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3ee15d-4073-44d1-b4db-2eb3edea6566-kube-api-access-q8m7p" pod "6b3ee15d-4073-44d1-b4db-2eb3edea6566" (UID: "6b3ee15d-4073-44d1-b4db-2eb3edea6566"). InnerVolumeSpecName "kube-api-access-q8m7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:15:01.722089 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:01.722052 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8m7p\" (UniqueName: \"kubernetes.io/projected/6b3ee15d-4073-44d1-b4db-2eb3edea6566-kube-api-access-q8m7p\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:15:01.722089 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:01.722088 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:15:01.722299 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:01.722102 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ee15d-4073-44d1-b4db-2eb3edea6566-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:15:02.414035 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:02.413997 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" Nov 03 10:15:02.414035 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:02.414019 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm" event={"ID":"6b3ee15d-4073-44d1-b4db-2eb3edea6566","Type":"ContainerDied","Data":"e097753dbd4ed48f6b5210893c82932bd789e8fbcf9846fc5df93a1429283f9c"} Nov 03 10:15:02.414331 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:02.414077 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e097753dbd4ed48f6b5210893c82932bd789e8fbcf9846fc5df93a1429283f9c" Nov 03 10:15:16.954291 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:15:16.954218 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:15:19.834646 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.834594 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6"] Nov 03 10:15:19.835242 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.835221 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6b3ee15d-4073-44d1-b4db-2eb3edea6566" containerName="revoke-keys" Nov 03 10:15:19.835353 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.835246 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3ee15d-4073-44d1-b4db-2eb3edea6566" containerName="revoke-keys" Nov 03 10:15:19.835413 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.835365 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3ee15d-4073-44d1-b4db-2eb3edea6566" containerName="revoke-keys" Nov 03 10:15:19.838820 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.838788 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:19.839134 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.839106 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6z6\" (UniqueName: \"kubernetes.io/projected/67c72e97-f877-462d-877b-a706f2b4eb4c-kube-api-access-gh6z6\") pod \"maas-api-revoke-sub-a4aef7d7-wvdv6\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:19.839356 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.839335 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvdv6\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:19.839513 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.839467 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvdv6\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:19.895789 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.895759 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6"] Nov 03 10:15:19.939974 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.939941 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvdv6\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:19.940148 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.940004 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6z6\" (UniqueName: \"kubernetes.io/projected/67c72e97-f877-462d-877b-a706f2b4eb4c-kube-api-access-gh6z6\") pod \"maas-api-revoke-sub-a4aef7d7-wvdv6\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:19.940148 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.940062 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvdv6\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:19.940646 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.940622 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvdv6\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:19.942283 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.942259 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:15:19.942424 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.942352 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:15:19.952679 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:19.952657 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvdv6\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:20.002653 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:20.002600 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6z6\" (UniqueName: \"kubernetes.io/projected/67c72e97-f877-462d-877b-a706f2b4eb4c-kube-api-access-gh6z6\") pod \"maas-api-revoke-sub-a4aef7d7-wvdv6\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:20.152688 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:20.152610 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:20.529752 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:20.529721 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6"] Nov 03 10:15:20.531036 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:15:20.531005 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c72e97_f877_462d_877b_a706f2b4eb4c.slice/crio-ea9901eb96c9552a2a5600d58f52cdfed9dba3ce8f629597d39c0b56d5710036 WatchSource:0}: Error finding container ea9901eb96c9552a2a5600d58f52cdfed9dba3ce8f629597d39c0b56d5710036: Status 404 returned error can't find the container with id ea9901eb96c9552a2a5600d58f52cdfed9dba3ce8f629597d39c0b56d5710036 Nov 03 10:15:20.546300 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:20.546201 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" event={"ID":"67c72e97-f877-462d-877b-a706f2b4eb4c","Type":"ContainerStarted","Data":"ea9901eb96c9552a2a5600d58f52cdfed9dba3ce8f629597d39c0b56d5710036"} Nov 03 10:15:21.554142 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:21.554106 5015 generic.go:356] "Generic (PLEG): container finished" podID="67c72e97-f877-462d-877b-a706f2b4eb4c" containerID="b51ca39ef98795761398bc8fec50452ebafabb6ea200f9c5671b0cb84a8b8109" exitCode=0 Nov 03 10:15:21.554564 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:21.554154 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" event={"ID":"67c72e97-f877-462d-877b-a706f2b4eb4c","Type":"ContainerDied","Data":"b51ca39ef98795761398bc8fec50452ebafabb6ea200f9c5671b0cb84a8b8109"} Nov 03 10:15:22.698951 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:22.698927 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:22.763241 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:22.763207 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-xks-ca\") pod \"67c72e97-f877-462d-877b-a706f2b4eb4c\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " Nov 03 10:15:22.763241 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:22.763247 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c72e97-f877-462d-877b-a706f2b4eb4c-kube-api-access-gh6z6\" (UniqueName: \"kubernetes.io/projected/67c72e97-f877-462d-877b-a706f2b4eb4c-kube-api-access-gh6z6\") pod \"67c72e97-f877-462d-877b-a706f2b4eb4c\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " Nov 03 10:15:22.763461 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:22.763368 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-openshift-ca\") pod \"67c72e97-f877-462d-877b-a706f2b4eb4c\" (UID: \"67c72e97-f877-462d-877b-a706f2b4eb4c\") " Nov 03 10:15:22.763741 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:22.763720 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-openshift-ca" pod "67c72e97-f877-462d-877b-a706f2b4eb4c" (UID: "67c72e97-f877-462d-877b-a706f2b4eb4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:15:22.765570 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:22.765546 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-xks-ca" pod "67c72e97-f877-462d-877b-a706f2b4eb4c" (UID: "67c72e97-f877-462d-877b-a706f2b4eb4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:15:22.765923 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:22.765902 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c72e97-f877-462d-877b-a706f2b4eb4c-kube-api-access-gh6z6" pod "67c72e97-f877-462d-877b-a706f2b4eb4c" (UID: "67c72e97-f877-462d-877b-a706f2b4eb4c"). InnerVolumeSpecName "kube-api-access-gh6z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:15:22.865093 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:22.864997 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:15:22.865093 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:22.865025 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c72e97-f877-462d-877b-a706f2b4eb4c-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:15:22.865093 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:22.865034 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh6z6\" (UniqueName: \"kubernetes.io/projected/67c72e97-f877-462d-877b-a706f2b4eb4c-kube-api-access-gh6z6\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:15:23.566880 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:23.566842 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" Nov 03 10:15:23.567064 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:23.566877 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6" event={"ID":"67c72e97-f877-462d-877b-a706f2b4eb4c","Type":"ContainerDied","Data":"ea9901eb96c9552a2a5600d58f52cdfed9dba3ce8f629597d39c0b56d5710036"} Nov 03 10:15:23.567064 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:23.566976 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea9901eb96c9552a2a5600d58f52cdfed9dba3ce8f629597d39c0b56d5710036" Nov 03 10:15:29.851969 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:29.851935 5015 scope.go:122] "RemoveContainer" containerID="3f9fb32e286f9de7a9649d4dd5754d888ba6b3e3fd309c8d253f598b056a71c0" Nov 03 10:15:29.862186 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:29.862158 5015 scope.go:122] "RemoveContainer" containerID="6ebff90b5d71113f960a15c4c0fa94823fe239d58f600440cef8631acb1fad6c" Nov 03 10:15:46.902999 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:15:46.902954 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:15:49.897227 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:49.896965 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv"] Nov 03 10:15:49.897744 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:49.897570 5015 cpu_manager.go:414] "containerMap: removing container" podUID="67c72e97-f877-462d-877b-a706f2b4eb4c" containerName="revoke-keys" Nov 03 10:15:49.897744 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:49.897585 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c72e97-f877-462d-877b-a706f2b4eb4c" containerName="revoke-keys" Nov 03 10:15:49.897744 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:49.897692 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c72e97-f877-462d-877b-a706f2b4eb4c" containerName="revoke-keys" Nov 03 10:15:49.900992 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:49.900973 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:49.957970 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:49.957935 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv"] Nov 03 10:15:49.994394 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:49.994350 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:15:50.004366 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.004336 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:15:50.063954 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.063910 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gqtjv\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:50.064168 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.063979 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqlw\" (UniqueName: \"kubernetes.io/projected/a608f782-0152-4296-ad6b-8ae268b6f4fb-kube-api-access-ztqlw\") pod \"maas-api-revoke-sub-79805ca9-gqtjv\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:50.064168 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.064145 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gqtjv\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:50.165239 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.165151 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqlw\" (UniqueName: \"kubernetes.io/projected/a608f782-0152-4296-ad6b-8ae268b6f4fb-kube-api-access-ztqlw\") pod \"maas-api-revoke-sub-79805ca9-gqtjv\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:50.165239 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.165228 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gqtjv\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:50.165436 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.165272 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gqtjv\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:50.165953 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.165917 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gqtjv\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:50.167904 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.167884 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gqtjv\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:50.224986 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.224953 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqlw\" (UniqueName: \"kubernetes.io/projected/a608f782-0152-4296-ad6b-8ae268b6f4fb-kube-api-access-ztqlw\") pod \"maas-api-revoke-sub-79805ca9-gqtjv\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:50.514310 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.514268 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:50.891592 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:50.891555 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv"] Nov 03 10:15:50.893420 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:15:50.893382 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda608f782_0152_4296_ad6b_8ae268b6f4fb.slice/crio-31317dfe485c30bfadb2c11cc2ad8c629236324db89ee766882ca1150cef0185 WatchSource:0}: Error finding container 31317dfe485c30bfadb2c11cc2ad8c629236324db89ee766882ca1150cef0185: Status 404 returned error can't find the container with id 31317dfe485c30bfadb2c11cc2ad8c629236324db89ee766882ca1150cef0185 Nov 03 10:15:51.790230 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:51.790196 5015 generic.go:356] "Generic (PLEG): container finished" podID="a608f782-0152-4296-ad6b-8ae268b6f4fb" containerID="be02ffa89a21c76abc9ca36702379f6d7f791042dda2b318624e41c8b956753d" exitCode=0 Nov 03 10:15:51.790666 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:51.790283 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" event={"ID":"a608f782-0152-4296-ad6b-8ae268b6f4fb","Type":"ContainerDied","Data":"be02ffa89a21c76abc9ca36702379f6d7f791042dda2b318624e41c8b956753d"} Nov 03 10:15:51.790666 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:51.790332 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" event={"ID":"a608f782-0152-4296-ad6b-8ae268b6f4fb","Type":"ContainerStarted","Data":"31317dfe485c30bfadb2c11cc2ad8c629236324db89ee766882ca1150cef0185"} Nov 03 10:15:52.935545 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:52.935519 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:53.097401 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.097322 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-openshift-ca\") pod \"a608f782-0152-4296-ad6b-8ae268b6f4fb\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " Nov 03 10:15:53.097601 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.097418 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-xks-ca\") pod \"a608f782-0152-4296-ad6b-8ae268b6f4fb\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " Nov 03 10:15:53.097601 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.097572 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a608f782-0152-4296-ad6b-8ae268b6f4fb-kube-api-access-ztqlw\" (UniqueName: \"kubernetes.io/projected/a608f782-0152-4296-ad6b-8ae268b6f4fb-kube-api-access-ztqlw\") pod \"a608f782-0152-4296-ad6b-8ae268b6f4fb\" (UID: \"a608f782-0152-4296-ad6b-8ae268b6f4fb\") " Nov 03 10:15:53.097782 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.097758 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-openshift-ca" pod "a608f782-0152-4296-ad6b-8ae268b6f4fb" (UID: "a608f782-0152-4296-ad6b-8ae268b6f4fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:15:53.097931 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.097908 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:15:53.099708 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.099679 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-xks-ca" pod "a608f782-0152-4296-ad6b-8ae268b6f4fb" (UID: "a608f782-0152-4296-ad6b-8ae268b6f4fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:15:53.099810 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.099753 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a608f782-0152-4296-ad6b-8ae268b6f4fb-kube-api-access-ztqlw" pod "a608f782-0152-4296-ad6b-8ae268b6f4fb" (UID: "a608f782-0152-4296-ad6b-8ae268b6f4fb"). InnerVolumeSpecName "kube-api-access-ztqlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:15:53.198314 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.198277 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqlw\" (UniqueName: \"kubernetes.io/projected/a608f782-0152-4296-ad6b-8ae268b6f4fb-kube-api-access-ztqlw\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:15:53.198314 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.198309 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a608f782-0152-4296-ad6b-8ae268b6f4fb-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:15:53.803437 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.803400 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" event={"ID":"a608f782-0152-4296-ad6b-8ae268b6f4fb","Type":"ContainerDied","Data":"31317dfe485c30bfadb2c11cc2ad8c629236324db89ee766882ca1150cef0185"} Nov 03 10:15:53.803642 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.803450 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv" Nov 03 10:15:53.803642 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:15:53.803455 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31317dfe485c30bfadb2c11cc2ad8c629236324db89ee766882ca1150cef0185" Nov 03 10:16:07.509744 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.509556 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr"] Nov 03 10:16:07.510318 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.510295 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a608f782-0152-4296-ad6b-8ae268b6f4fb" containerName="revoke-keys" Nov 03 10:16:07.510397 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.510317 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a608f782-0152-4296-ad6b-8ae268b6f4fb" containerName="revoke-keys" Nov 03 10:16:07.510456 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.510427 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a608f782-0152-4296-ad6b-8ae268b6f4fb" containerName="revoke-keys" Nov 03 10:16:07.514564 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.514437 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:07.558501 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.558048 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-t55jr\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:07.558501 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.558104 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7md8\" (UniqueName: \"kubernetes.io/projected/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-kube-api-access-b7md8\") pod \"maas-api-revoke-sub-0b7e8edb-t55jr\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:07.558501 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.558163 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-t55jr\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:07.616022 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.615905 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:16:07.616022 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.615990 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:16:07.626872 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.626842 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr"] Nov 03 10:16:07.724901 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.724105 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-t55jr\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:07.724901 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.724161 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7md8\" (UniqueName: \"kubernetes.io/projected/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-kube-api-access-b7md8\") pod \"maas-api-revoke-sub-0b7e8edb-t55jr\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:07.724901 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.724216 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-t55jr\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:07.725447 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.725193 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-t55jr\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:07.728409 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.728379 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-t55jr\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:07.803594 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.803506 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7md8\" (UniqueName: \"kubernetes.io/projected/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-kube-api-access-b7md8\") pod \"maas-api-revoke-sub-0b7e8edb-t55jr\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:07.829083 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:07.829048 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:08.244723 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:08.244682 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr"] Nov 03 10:16:08.245913 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:16:08.245890 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0aed2a_9a79_4179_8ef0_059f1d5255ce.slice/crio-5186ac2f9473ac4e6ce7acf496097518c2ec4d4bd7c43ea208422d28241b25d2 WatchSource:0}: Error finding container 5186ac2f9473ac4e6ce7acf496097518c2ec4d4bd7c43ea208422d28241b25d2: Status 404 returned error can't find the container with id 5186ac2f9473ac4e6ce7acf496097518c2ec4d4bd7c43ea208422d28241b25d2 Nov 03 10:16:08.970368 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:08.970260 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" event={"ID":"0e0aed2a-9a79-4179-8ef0-059f1d5255ce","Type":"ContainerStarted","Data":"226bcffa62adc569b1d3b3198e1630f29f47aa55152e37c4d6db0d4ef1b79d3c"} Nov 03 10:16:08.970368 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:08.970335 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" event={"ID":"0e0aed2a-9a79-4179-8ef0-059f1d5255ce","Type":"ContainerStarted","Data":"5186ac2f9473ac4e6ce7acf496097518c2ec4d4bd7c43ea208422d28241b25d2"} Nov 03 10:16:09.088196 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:09.088146 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" podStartSLOduration=2.088131677 podStartE2EDuration="2.088131677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:16:09.086802768 +0000 UTC m=+5802.011744978" watchObservedRunningTime="2026-11-03 10:16:09.088131677 +0000 UTC m=+5802.013073899" Nov 03 10:16:09.977714 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:09.977679 5015 generic.go:356] "Generic (PLEG): container finished" podID="0e0aed2a-9a79-4179-8ef0-059f1d5255ce" containerID="226bcffa62adc569b1d3b3198e1630f29f47aa55152e37c4d6db0d4ef1b79d3c" exitCode=0 Nov 03 10:16:09.978123 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:09.977779 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" event={"ID":"0e0aed2a-9a79-4179-8ef0-059f1d5255ce","Type":"ContainerDied","Data":"226bcffa62adc569b1d3b3198e1630f29f47aa55152e37c4d6db0d4ef1b79d3c"} Nov 03 10:16:11.131057 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.131031 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:11.261844 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.261773 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-xks-ca\") pod \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " Nov 03 10:16:11.261844 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.261833 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-kube-api-access-b7md8\" (UniqueName: \"kubernetes.io/projected/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-kube-api-access-b7md8\") pod \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " Nov 03 10:16:11.262038 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.261859 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-openshift-ca\") pod \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\" (UID: \"0e0aed2a-9a79-4179-8ef0-059f1d5255ce\") " Nov 03 10:16:11.262280 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.262255 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-openshift-ca" pod "0e0aed2a-9a79-4179-8ef0-059f1d5255ce" (UID: "0e0aed2a-9a79-4179-8ef0-059f1d5255ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:16:11.264268 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.264238 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-xks-ca" pod "0e0aed2a-9a79-4179-8ef0-059f1d5255ce" (UID: "0e0aed2a-9a79-4179-8ef0-059f1d5255ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:16:11.264385 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.264268 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-kube-api-access-b7md8" pod "0e0aed2a-9a79-4179-8ef0-059f1d5255ce" (UID: "0e0aed2a-9a79-4179-8ef0-059f1d5255ce"). InnerVolumeSpecName "kube-api-access-b7md8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:16:11.362830 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.362798 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:16:11.362830 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.362827 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7md8\" (UniqueName: \"kubernetes.io/projected/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-kube-api-access-b7md8\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:16:11.363040 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.362838 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e0aed2a-9a79-4179-8ef0-059f1d5255ce-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:16:11.991874 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.991845 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" Nov 03 10:16:11.992081 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.991891 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr" event={"ID":"0e0aed2a-9a79-4179-8ef0-059f1d5255ce","Type":"ContainerDied","Data":"5186ac2f9473ac4e6ce7acf496097518c2ec4d4bd7c43ea208422d28241b25d2"} Nov 03 10:16:11.992081 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:11.991948 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5186ac2f9473ac4e6ce7acf496097518c2ec4d4bd7c43ea208422d28241b25d2" Nov 03 10:16:16.902628 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:16:16.902588 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:16:18.818769 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:18.818730 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq"] Nov 03 10:16:18.820389 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:18.820359 5015 cpu_manager.go:414] "containerMap: removing container" podUID="0e0aed2a-9a79-4179-8ef0-059f1d5255ce" containerName="revoke-keys" Nov 03 10:16:18.820584 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:18.820569 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0aed2a-9a79-4179-8ef0-059f1d5255ce" containerName="revoke-keys" Nov 03 10:16:18.820941 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:18.820925 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0aed2a-9a79-4179-8ef0-059f1d5255ce" containerName="revoke-keys" Nov 03 10:16:18.825268 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:18.825240 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:18.877555 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:18.877521 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq"] Nov 03 10:16:18.880050 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:18.880024 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:16:18.880218 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:18.880092 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:16:18.925743 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:18.925707 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p9pcq\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:18.925743 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:18.925747 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbgt\" (UniqueName: \"kubernetes.io/projected/7db87be9-629d-4edc-8895-202f06cb06cf-kube-api-access-kgbgt\") pod \"maas-api-revoke-sub-3d4156f1-p9pcq\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:18.926018 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:18.925771 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p9pcq\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:19.027369 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:19.027324 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p9pcq\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:19.027369 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:19.027369 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbgt\" (UniqueName: \"kubernetes.io/projected/7db87be9-629d-4edc-8895-202f06cb06cf-kube-api-access-kgbgt\") pod \"maas-api-revoke-sub-3d4156f1-p9pcq\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:19.027662 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:19.027404 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p9pcq\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:19.028134 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:19.028106 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p9pcq\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:19.030028 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:19.030004 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p9pcq\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:19.086957 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:19.086877 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbgt\" (UniqueName: \"kubernetes.io/projected/7db87be9-629d-4edc-8895-202f06cb06cf-kube-api-access-kgbgt\") pod \"maas-api-revoke-sub-3d4156f1-p9pcq\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:19.144911 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:19.144878 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:19.522117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:19.522081 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq"] Nov 03 10:16:19.525897 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:16:19.524795 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db87be9_629d_4edc_8895_202f06cb06cf.slice/crio-4bcac7d6a704cfce07955b02d6470df44cf02d72b11b37998077ef4206ae5aab WatchSource:0}: Error finding container 4bcac7d6a704cfce07955b02d6470df44cf02d72b11b37998077ef4206ae5aab: Status 404 returned error can't find the container with id 4bcac7d6a704cfce07955b02d6470df44cf02d72b11b37998077ef4206ae5aab Nov 03 10:16:20.042805 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:20.042773 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" event={"ID":"7db87be9-629d-4edc-8895-202f06cb06cf","Type":"ContainerStarted","Data":"882c403704725c1f05b1cbfc9ee9b7b43c75fbf02fba3cd90a5d110ea2290312"} Nov 03 10:16:20.043224 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:20.042829 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" event={"ID":"7db87be9-629d-4edc-8895-202f06cb06cf","Type":"ContainerStarted","Data":"4bcac7d6a704cfce07955b02d6470df44cf02d72b11b37998077ef4206ae5aab"} Nov 03 10:16:20.160618 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:20.160493 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" podStartSLOduration=2.160460542 podStartE2EDuration="2.160460542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:16:20.157385388 +0000 UTC m=+5813.082327624" watchObservedRunningTime="2026-11-03 10:16:20.160460542 +0000 UTC m=+5813.085402789" Nov 03 10:16:21.051795 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:21.051762 5015 generic.go:356] "Generic (PLEG): container finished" podID="7db87be9-629d-4edc-8895-202f06cb06cf" containerID="882c403704725c1f05b1cbfc9ee9b7b43c75fbf02fba3cd90a5d110ea2290312" exitCode=0 Nov 03 10:16:21.052305 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:21.051863 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" event={"ID":"7db87be9-629d-4edc-8895-202f06cb06cf","Type":"ContainerDied","Data":"882c403704725c1f05b1cbfc9ee9b7b43c75fbf02fba3cd90a5d110ea2290312"} Nov 03 10:16:22.196818 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:22.196790 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:22.262351 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:22.262314 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-openshift-ca\") pod \"7db87be9-629d-4edc-8895-202f06cb06cf\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " Nov 03 10:16:22.262585 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:22.262371 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-xks-ca\") pod \"7db87be9-629d-4edc-8895-202f06cb06cf\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " Nov 03 10:16:22.262585 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:22.262514 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db87be9-629d-4edc-8895-202f06cb06cf-kube-api-access-kgbgt\" (UniqueName: \"kubernetes.io/projected/7db87be9-629d-4edc-8895-202f06cb06cf-kube-api-access-kgbgt\") pod \"7db87be9-629d-4edc-8895-202f06cb06cf\" (UID: \"7db87be9-629d-4edc-8895-202f06cb06cf\") " Nov 03 10:16:22.262815 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:22.262792 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-openshift-ca" pod "7db87be9-629d-4edc-8895-202f06cb06cf" (UID: "7db87be9-629d-4edc-8895-202f06cb06cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:16:22.262952 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:22.262933 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:16:22.265310 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:22.265288 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-xks-ca" pod "7db87be9-629d-4edc-8895-202f06cb06cf" (UID: "7db87be9-629d-4edc-8895-202f06cb06cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:16:22.270056 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:22.270026 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db87be9-629d-4edc-8895-202f06cb06cf-kube-api-access-kgbgt" pod "7db87be9-629d-4edc-8895-202f06cb06cf" (UID: "7db87be9-629d-4edc-8895-202f06cb06cf"). InnerVolumeSpecName "kube-api-access-kgbgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:16:22.364279 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:22.364184 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db87be9-629d-4edc-8895-202f06cb06cf-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:16:22.364279 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:22.364224 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgbgt\" (UniqueName: \"kubernetes.io/projected/7db87be9-629d-4edc-8895-202f06cb06cf-kube-api-access-kgbgt\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:16:23.065972 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:23.065934 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" Nov 03 10:16:23.065972 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:23.065963 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq" event={"ID":"7db87be9-629d-4edc-8895-202f06cb06cf","Type":"ContainerDied","Data":"4bcac7d6a704cfce07955b02d6470df44cf02d72b11b37998077ef4206ae5aab"} Nov 03 10:16:23.066260 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:23.066012 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bcac7d6a704cfce07955b02d6470df44cf02d72b11b37998077ef4206ae5aab" Nov 03 10:16:29.889528 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:29.889468 5015 scope.go:122] "RemoveContainer" containerID="4d663b23fa57f20922fe25b5d5b1601da19299e6b999af852561cdbe7f1e5e8e" Nov 03 10:16:37.356190 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.356156 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq"] Nov 03 10:16:37.406459 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.406428 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p9pcq"] Nov 03 10:16:37.481849 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.481817 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx"] Nov 03 10:16:37.482253 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.482241 5015 cpu_manager.go:414] "containerMap: removing container" podUID="7db87be9-629d-4edc-8895-202f06cb06cf" containerName="revoke-keys" Nov 03 10:16:37.482298 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.482254 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db87be9-629d-4edc-8895-202f06cb06cf" containerName="revoke-keys" Nov 03 10:16:37.482363 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.482354 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db87be9-629d-4edc-8895-202f06cb06cf" containerName="revoke-keys" Nov 03 10:16:37.485516 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.485495 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:37.537821 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.537786 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s9k5\" (UniqueName: \"kubernetes.io/projected/2f6564a9-de1e-4d93-b767-844e2478b6be-kube-api-access-9s9k5\") pod \"maas-api-revoke-sub-3d4156f1-hmdxx\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:37.537991 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.537911 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hmdxx\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:37.538044 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.537998 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hmdxx\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:37.540396 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.540332 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:16:37.540683 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.540663 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:16:37.542660 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.542637 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx"] Nov 03 10:16:37.639025 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.638935 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hmdxx\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:37.639025 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.638985 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s9k5\" (UniqueName: \"kubernetes.io/projected/2f6564a9-de1e-4d93-b767-844e2478b6be-kube-api-access-9s9k5\") pod \"maas-api-revoke-sub-3d4156f1-hmdxx\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:37.639282 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.639032 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hmdxx\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:37.639678 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.639658 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hmdxx\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:37.641590 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.641572 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hmdxx\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:37.698444 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.698415 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s9k5\" (UniqueName: \"kubernetes.io/projected/2f6564a9-de1e-4d93-b767-844e2478b6be-kube-api-access-9s9k5\") pod \"maas-api-revoke-sub-3d4156f1-hmdxx\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:37.798149 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:37.798114 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:38.178549 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:16:38.178466 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6564a9_de1e_4d93_b767_844e2478b6be.slice/crio-7f495dc8a18542230e1fdfa343b360e78fa91245fc94b520cf7b25f087adcca2 WatchSource:0}: Error finding container 7f495dc8a18542230e1fdfa343b360e78fa91245fc94b520cf7b25f087adcca2: Status 404 returned error can't find the container with id 7f495dc8a18542230e1fdfa343b360e78fa91245fc94b520cf7b25f087adcca2 Nov 03 10:16:38.179516 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:38.179491 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx"] Nov 03 10:16:38.223796 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:38.223765 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" event={"ID":"2f6564a9-de1e-4d93-b767-844e2478b6be","Type":"ContainerStarted","Data":"7f495dc8a18542230e1fdfa343b360e78fa91245fc94b520cf7b25f087adcca2"} Nov 03 10:16:39.203629 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:39.203596 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db87be9-629d-4edc-8895-202f06cb06cf" path="/var/lib/kubelet/pods/7db87be9-629d-4edc-8895-202f06cb06cf/volumes" Nov 03 10:16:39.234559 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:39.234525 5015 generic.go:356] "Generic (PLEG): container finished" podID="2f6564a9-de1e-4d93-b767-844e2478b6be" containerID="231f1e493095d949e56cdd5d21857af1a639330ccb75752ea6a1dd08757364d5" exitCode=0 Nov 03 10:16:39.234728 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:39.234615 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" event={"ID":"2f6564a9-de1e-4d93-b767-844e2478b6be","Type":"ContainerDied","Data":"231f1e493095d949e56cdd5d21857af1a639330ccb75752ea6a1dd08757364d5"} Nov 03 10:16:40.378397 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:40.378370 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:40.466829 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:40.466792 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-openshift-ca\") pod \"2f6564a9-de1e-4d93-b767-844e2478b6be\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " Nov 03 10:16:40.467047 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:40.466864 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-xks-ca\") pod \"2f6564a9-de1e-4d93-b767-844e2478b6be\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " Nov 03 10:16:40.467047 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:40.466977 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6564a9-de1e-4d93-b767-844e2478b6be-kube-api-access-9s9k5\" (UniqueName: \"kubernetes.io/projected/2f6564a9-de1e-4d93-b767-844e2478b6be-kube-api-access-9s9k5\") pod \"2f6564a9-de1e-4d93-b767-844e2478b6be\" (UID: \"2f6564a9-de1e-4d93-b767-844e2478b6be\") " Nov 03 10:16:40.467265 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:40.467243 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-openshift-ca" pod "2f6564a9-de1e-4d93-b767-844e2478b6be" (UID: "2f6564a9-de1e-4d93-b767-844e2478b6be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:16:40.467381 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:40.467363 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:16:40.469356 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:40.469329 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-xks-ca" pod "2f6564a9-de1e-4d93-b767-844e2478b6be" (UID: "2f6564a9-de1e-4d93-b767-844e2478b6be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:16:40.469356 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:40.469347 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6564a9-de1e-4d93-b767-844e2478b6be-kube-api-access-9s9k5" pod "2f6564a9-de1e-4d93-b767-844e2478b6be" (UID: "2f6564a9-de1e-4d93-b767-844e2478b6be"). InnerVolumeSpecName "kube-api-access-9s9k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:16:40.568175 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:40.568089 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6564a9-de1e-4d93-b767-844e2478b6be-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:16:40.568175 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:40.568126 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s9k5\" (UniqueName: \"kubernetes.io/projected/2f6564a9-de1e-4d93-b767-844e2478b6be-kube-api-access-9s9k5\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:16:41.248343 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:41.248312 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" Nov 03 10:16:41.248569 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:41.248345 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx" event={"ID":"2f6564a9-de1e-4d93-b767-844e2478b6be","Type":"ContainerDied","Data":"7f495dc8a18542230e1fdfa343b360e78fa91245fc94b520cf7b25f087adcca2"} Nov 03 10:16:41.248569 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:16:41.248389 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f495dc8a18542230e1fdfa343b360e78fa91245fc94b520cf7b25f087adcca2" Nov 03 10:16:46.901029 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:16:46.900991 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:17:09.053511 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:09.053435 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf"] Nov 03 10:17:09.113572 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:09.113532 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-zt9vf"] Nov 03 10:17:09.203494 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:09.203446 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e74267-eff8-4a6e-b04d-0346a5cd233c" path="/var/lib/kubelet/pods/d0e74267-eff8-4a6e-b04d-0346a5cd233c/volumes" Nov 03 10:17:12.007078 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.007040 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn"] Nov 03 10:17:12.007553 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.007524 5015 cpu_manager.go:414] "containerMap: removing container" podUID="2f6564a9-de1e-4d93-b767-844e2478b6be" containerName="revoke-keys" Nov 03 10:17:12.007553 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.007537 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6564a9-de1e-4d93-b767-844e2478b6be" containerName="revoke-keys" Nov 03 10:17:12.007653 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.007620 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6564a9-de1e-4d93-b767-844e2478b6be" containerName="revoke-keys" Nov 03 10:17:12.012057 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.012035 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:12.067704 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.067665 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn"] Nov 03 10:17:12.109024 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.108986 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc47b\" (UniqueName: \"kubernetes.io/projected/becc83f1-c9c5-40da-bac7-d529245cf0ac-kube-api-access-qc47b\") pod \"maas-api-revoke-sub-e2fc9c5b-sw2jn\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:12.109245 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.109134 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sw2jn\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:12.109245 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.109226 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sw2jn\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:12.114277 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.114252 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:17:12.114277 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.114259 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:17:12.209973 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.209930 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc47b\" (UniqueName: \"kubernetes.io/projected/becc83f1-c9c5-40da-bac7-d529245cf0ac-kube-api-access-qc47b\") pod \"maas-api-revoke-sub-e2fc9c5b-sw2jn\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:12.210187 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.210054 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sw2jn\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:12.210187 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.210094 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sw2jn\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:12.210808 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.210787 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sw2jn\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:12.212762 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.212734 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sw2jn\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:12.270235 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.270152 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc47b\" (UniqueName: \"kubernetes.io/projected/becc83f1-c9c5-40da-bac7-d529245cf0ac-kube-api-access-qc47b\") pod \"maas-api-revoke-sub-e2fc9c5b-sw2jn\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:12.325550 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.325462 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:12.706845 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:12.706816 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn"] Nov 03 10:17:12.708378 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:17:12.708353 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecc83f1_c9c5_40da_bac7_d529245cf0ac.slice/crio-d6b65180308020dacdd37913196e3392ed7a4ee4a83b2f9e97fee9bfbc68f977 WatchSource:0}: Error finding container d6b65180308020dacdd37913196e3392ed7a4ee4a83b2f9e97fee9bfbc68f977: Status 404 returned error can't find the container with id d6b65180308020dacdd37913196e3392ed7a4ee4a83b2f9e97fee9bfbc68f977 Nov 03 10:17:13.465165 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:13.465070 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" event={"ID":"becc83f1-c9c5-40da-bac7-d529245cf0ac","Type":"ContainerStarted","Data":"8f212e7aaa6ef1bd9499d84230e74e0873fcf05cfaf4018037dffb1b2fadd248"} Nov 03 10:17:13.465165 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:13.465130 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" event={"ID":"becc83f1-c9c5-40da-bac7-d529245cf0ac","Type":"ContainerStarted","Data":"d6b65180308020dacdd37913196e3392ed7a4ee4a83b2f9e97fee9bfbc68f977"} Nov 03 10:17:13.584497 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:13.584425 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" podStartSLOduration=2.584409855 podStartE2EDuration="2.584409855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:17:13.581073446 +0000 UTC m=+5866.506015664" watchObservedRunningTime="2026-11-03 10:17:13.584409855 +0000 UTC m=+5866.509352072" Nov 03 10:17:14.476593 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:14.476548 5015 generic.go:356] "Generic (PLEG): container finished" podID="becc83f1-c9c5-40da-bac7-d529245cf0ac" containerID="8f212e7aaa6ef1bd9499d84230e74e0873fcf05cfaf4018037dffb1b2fadd248" exitCode=0 Nov 03 10:17:14.477037 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:14.476622 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" event={"ID":"becc83f1-c9c5-40da-bac7-d529245cf0ac","Type":"ContainerDied","Data":"8f212e7aaa6ef1bd9499d84230e74e0873fcf05cfaf4018037dffb1b2fadd248"} Nov 03 10:17:15.629509 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:15.629447 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:15.745170 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:15.745078 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/becc83f1-c9c5-40da-bac7-d529245cf0ac-kube-api-access-qc47b\" (UniqueName: \"kubernetes.io/projected/becc83f1-c9c5-40da-bac7-d529245cf0ac-kube-api-access-qc47b\") pod \"becc83f1-c9c5-40da-bac7-d529245cf0ac\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " Nov 03 10:17:15.745542 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:15.745519 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-openshift-ca\") pod \"becc83f1-c9c5-40da-bac7-d529245cf0ac\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " Nov 03 10:17:15.745723 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:15.745707 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-xks-ca\") pod \"becc83f1-c9c5-40da-bac7-d529245cf0ac\" (UID: \"becc83f1-c9c5-40da-bac7-d529245cf0ac\") " Nov 03 10:17:15.746054 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:15.745970 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-openshift-ca" pod "becc83f1-c9c5-40da-bac7-d529245cf0ac" (UID: "becc83f1-c9c5-40da-bac7-d529245cf0ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:17:15.746245 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:15.746177 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:17:15.748707 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:15.748681 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-xks-ca" pod "becc83f1-c9c5-40da-bac7-d529245cf0ac" (UID: "becc83f1-c9c5-40da-bac7-d529245cf0ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:17:15.749628 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:15.749603 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becc83f1-c9c5-40da-bac7-d529245cf0ac-kube-api-access-qc47b" pod "becc83f1-c9c5-40da-bac7-d529245cf0ac" (UID: "becc83f1-c9c5-40da-bac7-d529245cf0ac"). InnerVolumeSpecName "kube-api-access-qc47b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:17:15.847788 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:15.847751 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc47b\" (UniqueName: \"kubernetes.io/projected/becc83f1-c9c5-40da-bac7-d529245cf0ac-kube-api-access-qc47b\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:17:15.847788 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:15.847783 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becc83f1-c9c5-40da-bac7-d529245cf0ac-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:17:16.490719 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:16.490685 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" Nov 03 10:17:16.490719 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:16.490711 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn" event={"ID":"becc83f1-c9c5-40da-bac7-d529245cf0ac","Type":"ContainerDied","Data":"d6b65180308020dacdd37913196e3392ed7a4ee4a83b2f9e97fee9bfbc68f977"} Nov 03 10:17:16.490950 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:16.490761 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6b65180308020dacdd37913196e3392ed7a4ee4a83b2f9e97fee9bfbc68f977" Nov 03 10:17:16.901970 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:17:16.901884 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:17:22.760100 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:22.760068 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h"] Nov 03 10:17:22.760520 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:22.760507 5015 cpu_manager.go:414] "containerMap: removing container" podUID="becc83f1-c9c5-40da-bac7-d529245cf0ac" containerName="revoke-keys" Nov 03 10:17:22.760572 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:22.760522 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="becc83f1-c9c5-40da-bac7-d529245cf0ac" containerName="revoke-keys" Nov 03 10:17:22.760620 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:22.760611 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="becc83f1-c9c5-40da-bac7-d529245cf0ac" containerName="revoke-keys" Nov 03 10:17:22.763870 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:22.763854 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:22.820972 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:22.820933 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h"] Nov 03 10:17:22.867113 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:22.867084 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:17:22.867298 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:22.867196 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:17:22.922267 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:22.922231 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-68z7h\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:22.922457 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:22.922351 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-68z7h\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:22.922457 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:22.922398 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7glx\" (UniqueName: \"kubernetes.io/projected/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-kube-api-access-k7glx\") pod \"maas-api-revoke-sub-5a29efb3-68z7h\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:23.023382 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:23.023291 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-68z7h\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:23.023382 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:23.023356 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-68z7h\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:23.023632 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:23.023389 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7glx\" (UniqueName: \"kubernetes.io/projected/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-kube-api-access-k7glx\") pod \"maas-api-revoke-sub-5a29efb3-68z7h\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:23.024218 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:23.024128 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-68z7h\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:23.026055 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:23.026033 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-68z7h\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:23.083651 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:23.083614 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7glx\" (UniqueName: \"kubernetes.io/projected/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-kube-api-access-k7glx\") pod \"maas-api-revoke-sub-5a29efb3-68z7h\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:23.376656 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:23.376566 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:23.755903 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:17:23.755861 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc106d3_e936_4ffa_978e_f0e1c4b09e36.slice/crio-f9ffaef0977e5c669f6087f8021c57c68db15cd3acb458836926e92eba4e929c WatchSource:0}: Error finding container f9ffaef0977e5c669f6087f8021c57c68db15cd3acb458836926e92eba4e929c: Status 404 returned error can't find the container with id f9ffaef0977e5c669f6087f8021c57c68db15cd3acb458836926e92eba4e929c Nov 03 10:17:23.757073 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:23.757042 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h"] Nov 03 10:17:24.543366 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:24.543330 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" event={"ID":"6fc106d3-e936-4ffa-978e-f0e1c4b09e36","Type":"ContainerStarted","Data":"9a7483bcb9ffc63b7d992af7a911367962c5f564325115fba997ccd4809fadb0"} Nov 03 10:17:24.544030 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:24.543393 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" event={"ID":"6fc106d3-e936-4ffa-978e-f0e1c4b09e36","Type":"ContainerStarted","Data":"f9ffaef0977e5c669f6087f8021c57c68db15cd3acb458836926e92eba4e929c"} Nov 03 10:17:24.661967 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:24.661909 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" podStartSLOduration=2.661894729 podStartE2EDuration="2.661894729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:17:24.658855287 +0000 UTC m=+5877.583797505" watchObservedRunningTime="2026-11-03 10:17:24.661894729 +0000 UTC m=+5877.586836949" Nov 03 10:17:25.553906 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:25.553865 5015 generic.go:356] "Generic (PLEG): container finished" podID="6fc106d3-e936-4ffa-978e-f0e1c4b09e36" containerID="9a7483bcb9ffc63b7d992af7a911367962c5f564325115fba997ccd4809fadb0" exitCode=0 Nov 03 10:17:25.554375 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:25.553967 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" event={"ID":"6fc106d3-e936-4ffa-978e-f0e1c4b09e36","Type":"ContainerDied","Data":"9a7483bcb9ffc63b7d992af7a911367962c5f564325115fba997ccd4809fadb0"} Nov 03 10:17:26.998129 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:26.997753 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:27.022986 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.022944 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-openshift-ca\") pod \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " Nov 03 10:17:27.023215 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.023048 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-kube-api-access-k7glx\" (UniqueName: \"kubernetes.io/projected/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-kube-api-access-k7glx\") pod \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " Nov 03 10:17:27.023215 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.023130 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-xks-ca\") pod \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\" (UID: \"6fc106d3-e936-4ffa-978e-f0e1c4b09e36\") " Nov 03 10:17:27.023795 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.023748 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-openshift-ca" pod "6fc106d3-e936-4ffa-978e-f0e1c4b09e36" (UID: "6fc106d3-e936-4ffa-978e-f0e1c4b09e36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:17:27.026623 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.026592 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-kube-api-access-k7glx" pod "6fc106d3-e936-4ffa-978e-f0e1c4b09e36" (UID: "6fc106d3-e936-4ffa-978e-f0e1c4b09e36"). InnerVolumeSpecName "kube-api-access-k7glx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:17:27.028604 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.028576 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-xks-ca" pod "6fc106d3-e936-4ffa-978e-f0e1c4b09e36" (UID: "6fc106d3-e936-4ffa-978e-f0e1c4b09e36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:17:27.124101 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.124057 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7glx\" (UniqueName: \"kubernetes.io/projected/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-kube-api-access-k7glx\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:17:27.124101 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.124102 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:17:27.124382 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.124118 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc106d3-e936-4ffa-978e-f0e1c4b09e36-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:17:27.623983 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.623889 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" event={"ID":"6fc106d3-e936-4ffa-978e-f0e1c4b09e36","Type":"ContainerDied","Data":"f9ffaef0977e5c669f6087f8021c57c68db15cd3acb458836926e92eba4e929c"} Nov 03 10:17:27.623983 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.623956 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9ffaef0977e5c669f6087f8021c57c68db15cd3acb458836926e92eba4e929c" Nov 03 10:17:27.623983 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:27.623908 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h" Nov 03 10:17:40.054196 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:40.054108 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz"] Nov 03 10:17:40.103785 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:40.103742 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7nwfz"] Nov 03 10:17:40.863830 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:40.863783 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx"] Nov 03 10:17:40.913907 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:40.913871 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hmdxx"] Nov 03 10:17:40.984382 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:40.984340 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw"] Nov 03 10:17:40.984900 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:40.984881 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6fc106d3-e936-4ffa-978e-f0e1c4b09e36" containerName="revoke-keys" Nov 03 10:17:40.985010 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:40.984902 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc106d3-e936-4ffa-978e-f0e1c4b09e36" containerName="revoke-keys" Nov 03 10:17:40.985071 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:40.985018 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc106d3-e936-4ffa-978e-f0e1c4b09e36" containerName="revoke-keys" Nov 03 10:17:40.989336 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:40.989315 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:41.045826 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.045797 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:17:41.046016 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.045947 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:17:41.049139 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.049113 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw"] Nov 03 10:17:41.070974 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.070929 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7brl\" (UniqueName: \"kubernetes.io/projected/5d0797dd-bcde-4c76-8e9f-7398f79aded4-kube-api-access-l7brl\") pod \"maas-api-revoke-sub-3d4156f1-f6ckw\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:41.071408 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.071079 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f6ckw\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:41.071408 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.071184 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f6ckw\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:41.172030 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.171939 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f6ckw\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:41.172030 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.172023 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7brl\" (UniqueName: \"kubernetes.io/projected/5d0797dd-bcde-4c76-8e9f-7398f79aded4-kube-api-access-l7brl\") pod \"maas-api-revoke-sub-3d4156f1-f6ckw\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:41.172282 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.172187 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f6ckw\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:41.172967 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.172931 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f6ckw\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:41.174642 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.174623 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f6ckw\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:41.203851 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.203816 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6564a9-de1e-4d93-b767-844e2478b6be" path="/var/lib/kubelet/pods/2f6564a9-de1e-4d93-b767-844e2478b6be/volumes" Nov 03 10:17:41.204244 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.204230 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319d5b19-0780-43a1-8980-42723cb1c38c" path="/var/lib/kubelet/pods/319d5b19-0780-43a1-8980-42723cb1c38c/volumes" Nov 03 10:17:41.231759 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.231729 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7brl\" (UniqueName: \"kubernetes.io/projected/5d0797dd-bcde-4c76-8e9f-7398f79aded4-kube-api-access-l7brl\") pod \"maas-api-revoke-sub-3d4156f1-f6ckw\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:41.302354 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.302311 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:41.682367 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.682336 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw"] Nov 03 10:17:41.683256 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:17:41.683229 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0797dd_bcde_4c76_8e9f_7398f79aded4.slice/crio-660fe631cd433f960894f75954fc664b87393648f0dcc07493fe49dc00839d7e WatchSource:0}: Error finding container 660fe631cd433f960894f75954fc664b87393648f0dcc07493fe49dc00839d7e: Status 404 returned error can't find the container with id 660fe631cd433f960894f75954fc664b87393648f0dcc07493fe49dc00839d7e Nov 03 10:17:41.719004 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:41.718971 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" event={"ID":"5d0797dd-bcde-4c76-8e9f-7398f79aded4","Type":"ContainerStarted","Data":"660fe631cd433f960894f75954fc664b87393648f0dcc07493fe49dc00839d7e"} Nov 03 10:17:42.726818 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:42.726783 5015 generic.go:356] "Generic (PLEG): container finished" podID="5d0797dd-bcde-4c76-8e9f-7398f79aded4" containerID="0fde7aa10420d6abaa59ab314a800a5a4e487a58d2772f158813822350577bb9" exitCode=0 Nov 03 10:17:42.727218 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:42.726870 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" event={"ID":"5d0797dd-bcde-4c76-8e9f-7398f79aded4","Type":"ContainerDied","Data":"0fde7aa10420d6abaa59ab314a800a5a4e487a58d2772f158813822350577bb9"} Nov 03 10:17:43.873318 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:43.873292 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:44.005112 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.005029 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-openshift-ca\") pod \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " Nov 03 10:17:44.005112 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.005102 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0797dd-bcde-4c76-8e9f-7398f79aded4-kube-api-access-l7brl\" (UniqueName: \"kubernetes.io/projected/5d0797dd-bcde-4c76-8e9f-7398f79aded4-kube-api-access-l7brl\") pod \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " Nov 03 10:17:44.005350 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.005172 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-xks-ca\") pod \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\" (UID: \"5d0797dd-bcde-4c76-8e9f-7398f79aded4\") " Nov 03 10:17:44.005556 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.005528 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-openshift-ca" pod "5d0797dd-bcde-4c76-8e9f-7398f79aded4" (UID: "5d0797dd-bcde-4c76-8e9f-7398f79aded4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:17:44.007422 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.007395 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-xks-ca" pod "5d0797dd-bcde-4c76-8e9f-7398f79aded4" (UID: "5d0797dd-bcde-4c76-8e9f-7398f79aded4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:17:44.007537 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.007522 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0797dd-bcde-4c76-8e9f-7398f79aded4-kube-api-access-l7brl" pod "5d0797dd-bcde-4c76-8e9f-7398f79aded4" (UID: "5d0797dd-bcde-4c76-8e9f-7398f79aded4"). InnerVolumeSpecName "kube-api-access-l7brl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:17:44.105932 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.105896 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7brl\" (UniqueName: \"kubernetes.io/projected/5d0797dd-bcde-4c76-8e9f-7398f79aded4-kube-api-access-l7brl\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:17:44.105932 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.105927 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:17:44.105932 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.105937 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0797dd-bcde-4c76-8e9f-7398f79aded4-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:17:44.741221 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.741192 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" Nov 03 10:17:44.741415 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.741234 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw" event={"ID":"5d0797dd-bcde-4c76-8e9f-7398f79aded4","Type":"ContainerDied","Data":"660fe631cd433f960894f75954fc664b87393648f0dcc07493fe49dc00839d7e"} Nov 03 10:17:44.741415 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:44.741284 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="660fe631cd433f960894f75954fc664b87393648f0dcc07493fe49dc00839d7e" Nov 03 10:17:46.901009 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:17:46.900973 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:17:57.556464 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.556429 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf"] Nov 03 10:17:57.556904 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.556894 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5d0797dd-bcde-4c76-8e9f-7398f79aded4" containerName="revoke-keys" Nov 03 10:17:57.556951 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.556906 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0797dd-bcde-4c76-8e9f-7398f79aded4" containerName="revoke-keys" Nov 03 10:17:57.557034 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.557023 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0797dd-bcde-4c76-8e9f-7398f79aded4" containerName="revoke-keys" Nov 03 10:17:57.560582 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.560565 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.614071 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.614040 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-65cb27-openshift-default-dockercfg-f29hj\"" Nov 03 10:17:57.668568 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.668535 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf"] Nov 03 10:17:57.737158 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.737120 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t85hs\" (UniqueName: \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-kube-api-access-t85hs\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.737158 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.737164 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-certs\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.737401 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.737185 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-podinfo\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.737401 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.737270 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-socket\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.737401 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.737392 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-credential-socket\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.737570 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.737414 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istiod-ca-cert\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.737570 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.737451 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-data\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.737670 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.737555 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-envoy\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.737734 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.737680 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-token\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.839033 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.838911 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-data\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.839033 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.838972 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-envoy\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.839324 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.839050 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-token\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.839324 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.839087 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t85hs\" (UniqueName: \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-kube-api-access-t85hs\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.839324 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.839125 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-certs\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.839324 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.839157 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-podinfo\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.839324 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.839187 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-socket\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.839324 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.839245 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-credential-socket\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.839324 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.839277 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istiod-ca-cert\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.839835 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.839796 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-data\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.840104 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.840073 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-certs\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.840218 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.840138 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istiod-ca-cert\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.840733 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.840711 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-socket\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.842857 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.842830 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-envoy\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.843499 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.843437 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-podinfo\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.843654 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.843634 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-credential-socket\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.899554 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.899517 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t85hs\" (UniqueName: \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-kube-api-access-t85hs\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:57.899669 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:57.899645 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-token\") pod \"e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:58.173854 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:58.173766 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:58.611872 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:58.611836 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf"] Nov 03 10:17:58.650332 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:17:58.650305 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eec8db9_80db_4ffa_acac_59dc5ad5d672.slice/crio-d31197edfb8e791362901e60044606f672f7df4287ea7e90285ad664c4cb24be WatchSource:0}: Error finding container d31197edfb8e791362901e60044606f672f7df4287ea7e90285ad664c4cb24be: Status 404 returned error can't find the container with id d31197edfb8e791362901e60044606f672f7df4287ea7e90285ad664c4cb24be Nov 03 10:17:58.652443 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:58.652411 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:17:58.652548 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:58.652499 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:17:58.652548 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:58.652544 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:17:58.832547 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:58.832505 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" event={"ID":"1eec8db9-80db-4ffa-acac-59dc5ad5d672","Type":"ContainerStarted","Data":"cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d"} Nov 03 10:17:58.832767 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:58.832573 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" event={"ID":"1eec8db9-80db-4ffa-acac-59dc5ad5d672","Type":"ContainerStarted","Data":"d31197edfb8e791362901e60044606f672f7df4287ea7e90285ad664c4cb24be"} Nov 03 10:17:58.955311 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:58.955200 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" podStartSLOduration=1.95518697 podStartE2EDuration="1.95518697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:17:58.954635823 +0000 UTC m=+5911.879578066" watchObservedRunningTime="2026-11-03 10:17:58.95518697 +0000 UTC m=+5911.880129187" Nov 03 10:17:59.174108 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:59.174073 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:59.179403 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:59.179379 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:59.846368 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:59.846335 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:17:59.847435 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:17:59.847406 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:18:05.058886 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.058852 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86"] Nov 03 10:18:05.084255 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.084213 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmp86"] Nov 03 10:18:05.203677 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.203644 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457fe3eb-abe4-4c84-b733-8977b9651b33" path="/var/lib/kubelet/pods/457fe3eb-abe4-4c84-b733-8977b9651b33/volumes" Nov 03 10:18:05.724168 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.724036 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk"] Nov 03 10:18:05.729766 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.729737 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:18:05.784144 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.784098 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk"] Nov 03 10:18:05.819375 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.819340 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk\" (UID: \"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:18:05.819588 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.819395 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9hhp\" (UniqueName: \"kubernetes.io/projected/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-kube-api-access-z9hhp\") pod \"payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk\" (UID: \"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:18:05.830095 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.830063 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-65cb27\"" Nov 03 10:18:05.830095 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.830071 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-dd3a13f6-dockercfg-cv49c\"" Nov 03 10:18:05.895405 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.895372 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr"] Nov 03 10:18:05.899333 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.899313 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:18:05.920106 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.920059 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9hhp\" (UniqueName: \"kubernetes.io/projected/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-kube-api-access-z9hhp\") pod \"payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk\" (UID: \"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:18:05.920247 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.920163 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkntb\" (UniqueName: \"kubernetes.io/projected/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-kube-api-access-mkntb\") pod \"payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr\" (UID: \"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:18:05.920348 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.920326 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk\" (UID: \"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:18:05.920399 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.920368 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr\" (UID: \"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:18:05.921088 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.921066 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk\" (UID: \"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:18:05.958700 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.958643 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr"] Nov 03 10:18:05.979512 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:05.979418 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9hhp\" (UniqueName: \"kubernetes.io/projected/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-kube-api-access-z9hhp\") pod \"payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk\" (UID: \"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:18:06.020981 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.020948 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr\" (UID: \"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:18:06.021162 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.021036 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkntb\" (UniqueName: \"kubernetes.io/projected/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-kube-api-access-mkntb\") pod \"payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr\" (UID: \"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:18:06.021916 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.021889 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr\" (UID: \"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:18:06.046432 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.046399 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:18:06.080658 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.080627 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkntb\" (UniqueName: \"kubernetes.io/projected/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-kube-api-access-mkntb\") pod \"payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr\" (UID: \"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:18:06.213505 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.213449 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:18:06.428187 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.428138 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk"] Nov 03 10:18:06.429576 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:18:06.429548 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be6bca8_d8a9_426e_98fe_a7cbfde87f0f.slice/crio-7c3e230021bc73189b44f16f9b68458d51ed050bcbc23a6023e9d5a26793c254 WatchSource:0}: Error finding container 7c3e230021bc73189b44f16f9b68458d51ed050bcbc23a6023e9d5a26793c254: Status 404 returned error can't find the container with id 7c3e230021bc73189b44f16f9b68458d51ed050bcbc23a6023e9d5a26793c254 Nov 03 10:18:06.592331 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.592302 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr"] Nov 03 10:18:06.593440 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:18:06.593413 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba4cb4c_e6f0_4397_99fd_caa92d757b0d.slice/crio-f3c6bdfaba865441914252188b7ca6f232ec8cbbadf26dfee83d0747fccd29a1 WatchSource:0}: Error finding container f3c6bdfaba865441914252188b7ca6f232ec8cbbadf26dfee83d0747fccd29a1: Status 404 returned error can't find the container with id f3c6bdfaba865441914252188b7ca6f232ec8cbbadf26dfee83d0747fccd29a1 Nov 03 10:18:06.901828 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.901710 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" event={"ID":"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d","Type":"ContainerStarted","Data":"e659342e3ddac2426c9c7bc5c5ff0d8d50dfa27cd354a6a4e393c5351d00b944"} Nov 03 10:18:06.901828 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.901814 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:18:06.902083 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.901837 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" event={"ID":"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d","Type":"ContainerStarted","Data":"f3c6bdfaba865441914252188b7ca6f232ec8cbbadf26dfee83d0747fccd29a1"} Nov 03 10:18:06.903503 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.903452 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" event={"ID":"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f","Type":"ContainerStarted","Data":"c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9"} Nov 03 10:18:06.903657 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.903521 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:18:06.903657 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:06.903543 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" event={"ID":"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f","Type":"ContainerStarted","Data":"7c3e230021bc73189b44f16f9b68458d51ed050bcbc23a6023e9d5a26793c254"} Nov 03 10:18:07.017867 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:07.017812 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" podStartSLOduration=2.017798673 podStartE2EDuration="2.017798673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:18:07.017467089 +0000 UTC m=+5919.942409307" watchObservedRunningTime="2026-11-03 10:18:07.017798673 +0000 UTC m=+5919.942740889" Nov 03 10:18:07.140523 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:07.136535 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" podStartSLOduration=2.136517216 podStartE2EDuration="2.136517216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:18:07.134279145 +0000 UTC m=+5920.059221364" watchObservedRunningTime="2026-11-03 10:18:07.136517216 +0000 UTC m=+5920.061459434" Nov 03 10:18:16.903542 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:18:16.903470 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:18:17.912267 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:17.912239 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:18:17.912805 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:17.912426 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:18:26.055403 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:26.055368 5015 generic.go:356] "Generic (PLEG): container finished" podID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerID="cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d" exitCode=0 Nov 03 10:18:26.055970 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:26.055445 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" event={"ID":"1eec8db9-80db-4ffa-acac-59dc5ad5d672","Type":"ContainerDied","Data":"cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d"} Nov 03 10:18:26.055970 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:26.055863 5015 scope.go:122] "RemoveContainer" containerID="cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d" Nov 03 10:18:26.057099 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:26.057070 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:18:26.057189 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:26.057144 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:18:26.057189 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:26.057174 5015 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 10:18:27.069008 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:27.068972 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" event={"ID":"1eec8db9-80db-4ffa-acac-59dc5ad5d672","Type":"ContainerStarted","Data":"6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a"} Nov 03 10:18:27.174066 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:27.174028 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:18:27.175301 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:27.175267 5015 patch_prober.go:29] interesting pod/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 03 10:18:27.175443 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:27.175311 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 03 10:18:28.062591 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:28.062546 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5"] Nov 03 10:18:28.108333 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:28.108298 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jtgs5"] Nov 03 10:18:28.174752 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:28.174718 5015 patch_prober.go:29] interesting pod/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 03 10:18:28.174938 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:28.174768 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 03 10:18:29.174447 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:29.174404 5015 patch_prober.go:29] interesting pod/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 03 10:18:29.174888 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:29.174497 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 03 10:18:29.203566 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:29.203516 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc66bb5-36ba-488e-8f4d-0dc347da4675" path="/var/lib/kubelet/pods/9dc66bb5-36ba-488e-8f4d-0dc347da4675/volumes" Nov 03 10:18:29.860524 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:29.858590 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:18:29.925548 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:29.925498 5015 scope.go:122] "RemoveContainer" containerID="eb4bc6d3d2ceccec7bcc771d22d5fa32c4cc83f7a672fb320d2cd4a57f44a4ae" Nov 03 10:18:30.175046 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:30.174960 5015 patch_prober.go:29] interesting pod/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 03 10:18:30.175046 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:30.175012 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 03 10:18:31.175085 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:31.175048 5015 patch_prober.go:29] interesting pod/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Nov 03 10:18:31.175533 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:31.175112 5015 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Nov 03 10:18:32.181077 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:32.181046 5015 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:18:32.182212 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:18:32.182189 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:18:46.903328 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:18:46.903287 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:19:13.053233 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:13.053149 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww"] Nov 03 10:19:13.103721 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:13.103687 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wscww"] Nov 03 10:19:13.203419 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:13.203387 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5c1f3a-07da-4d3c-8428-e06f6647b23a" path="/var/lib/kubelet/pods/0a5c1f3a-07da-4d3c-8428-e06f6647b23a/volumes" Nov 03 10:19:15.562174 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.562144 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs"] Nov 03 10:19:15.565960 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.565944 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:15.621917 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.621877 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r27d\" (UniqueName: \"kubernetes.io/projected/6d3afce2-7eb1-43c8-9d84-b912000d7063-kube-api-access-2r27d\") pod \"maas-api-revoke-sub-b553a7a9-88qxs\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:15.622165 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.621950 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b553a7a9-88qxs\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:15.622238 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.622180 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b553a7a9-88qxs\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:15.623715 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.623692 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs"] Nov 03 10:19:15.670275 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.670235 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:19:15.670458 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.670300 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:19:15.722742 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.722709 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b553a7a9-88qxs\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:15.722935 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.722761 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r27d\" (UniqueName: \"kubernetes.io/projected/6d3afce2-7eb1-43c8-9d84-b912000d7063-kube-api-access-2r27d\") pod \"maas-api-revoke-sub-b553a7a9-88qxs\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:15.722935 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.722794 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b553a7a9-88qxs\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:15.723493 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.723444 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b553a7a9-88qxs\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:15.725284 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.725260 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b553a7a9-88qxs\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:15.781813 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.781781 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r27d\" (UniqueName: \"kubernetes.io/projected/6d3afce2-7eb1-43c8-9d84-b912000d7063-kube-api-access-2r27d\") pod \"maas-api-revoke-sub-b553a7a9-88qxs\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:15.880215 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:15.880122 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:16.260895 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:16.260857 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs"] Nov 03 10:19:16.261946 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:19:16.261921 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3afce2_7eb1_43c8_9d84_b912000d7063.slice/crio-b8f5e1dafd74d3ecf05e453ba4ab3466d62475a205eadcd3b9240e149cb627e7 WatchSource:0}: Error finding container b8f5e1dafd74d3ecf05e453ba4ab3466d62475a205eadcd3b9240e149cb627e7: Status 404 returned error can't find the container with id b8f5e1dafd74d3ecf05e453ba4ab3466d62475a205eadcd3b9240e149cb627e7 Nov 03 10:19:16.409136 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:16.409098 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" event={"ID":"6d3afce2-7eb1-43c8-9d84-b912000d7063","Type":"ContainerStarted","Data":"b8f5e1dafd74d3ecf05e453ba4ab3466d62475a205eadcd3b9240e149cb627e7"} Nov 03 10:19:16.902287 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:19:16.902202 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:19:17.417064 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:17.417029 5015 generic.go:356] "Generic (PLEG): container finished" podID="6d3afce2-7eb1-43c8-9d84-b912000d7063" containerID="abade9a71f62b68c97f122e45770898f6bbeaa43611fb385f4ac6b1eb7deaa98" exitCode=0 Nov 03 10:19:17.417279 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:17.417120 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" event={"ID":"6d3afce2-7eb1-43c8-9d84-b912000d7063","Type":"ContainerDied","Data":"abade9a71f62b68c97f122e45770898f6bbeaa43611fb385f4ac6b1eb7deaa98"} Nov 03 10:19:18.576052 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:18.576027 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:18.652787 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:18.652756 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-xks-ca\") pod \"6d3afce2-7eb1-43c8-9d84-b912000d7063\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " Nov 03 10:19:18.652963 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:18.652867 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-openshift-ca\") pod \"6d3afce2-7eb1-43c8-9d84-b912000d7063\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " Nov 03 10:19:18.652963 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:18.652891 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3afce2-7eb1-43c8-9d84-b912000d7063-kube-api-access-2r27d\" (UniqueName: \"kubernetes.io/projected/6d3afce2-7eb1-43c8-9d84-b912000d7063-kube-api-access-2r27d\") pod \"6d3afce2-7eb1-43c8-9d84-b912000d7063\" (UID: \"6d3afce2-7eb1-43c8-9d84-b912000d7063\") " Nov 03 10:19:18.653284 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:18.653255 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-openshift-ca" pod "6d3afce2-7eb1-43c8-9d84-b912000d7063" (UID: "6d3afce2-7eb1-43c8-9d84-b912000d7063"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:19:18.655219 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:18.655187 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-xks-ca" pod "6d3afce2-7eb1-43c8-9d84-b912000d7063" (UID: "6d3afce2-7eb1-43c8-9d84-b912000d7063"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:19:18.655339 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:18.655322 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3afce2-7eb1-43c8-9d84-b912000d7063-kube-api-access-2r27d" pod "6d3afce2-7eb1-43c8-9d84-b912000d7063" (UID: "6d3afce2-7eb1-43c8-9d84-b912000d7063"). InnerVolumeSpecName "kube-api-access-2r27d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:19:18.754915 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:18.754875 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:19:18.754915 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:18.754918 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3afce2-7eb1-43c8-9d84-b912000d7063-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:19:18.755221 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:18.754936 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r27d\" (UniqueName: \"kubernetes.io/projected/6d3afce2-7eb1-43c8-9d84-b912000d7063-kube-api-access-2r27d\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:19:19.431727 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:19.431690 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" event={"ID":"6d3afce2-7eb1-43c8-9d84-b912000d7063","Type":"ContainerDied","Data":"b8f5e1dafd74d3ecf05e453ba4ab3466d62475a205eadcd3b9240e149cb627e7"} Nov 03 10:19:19.431727 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:19.431722 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs" Nov 03 10:19:19.431957 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:19.431747 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f5e1dafd74d3ecf05e453ba4ab3466d62475a205eadcd3b9240e149cb627e7" Nov 03 10:19:24.052590 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:24.052551 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq"] Nov 03 10:19:24.102833 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:24.102799 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-s2fgq"] Nov 03 10:19:25.203353 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:25.203316 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2cd64a-ff55-48d1-9372-6cc05c6d43dd" path="/var/lib/kubelet/pods/be2cd64a-ff55-48d1-9372-6cc05c6d43dd/volumes" Nov 03 10:19:28.318940 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:28.318912 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:19:28.343026 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:28.343002 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:19:29.980777 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:29.980744 5015 scope.go:122] "RemoveContainer" containerID="3abae6dac360af050208a9e09d5ac208907d7147b72b729d68d960b76118080c" Nov 03 10:19:29.992122 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:29.992100 5015 scope.go:122] "RemoveContainer" containerID="58303066e93f2bcd5bd0a3708e6aa182a7f1f5c9f56a8c8977a04cbac893e036" Nov 03 10:19:30.001311 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:30.001290 5015 scope.go:122] "RemoveContainer" containerID="0d029d4614a22297e48f0976230e7315a9b625b8e34ed04ec573d46d4c0d1817" Nov 03 10:19:36.448665 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:36.448625 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk"] Nov 03 10:19:36.557442 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:36.557405 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr"] Nov 03 10:19:41.449875 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:41.449817 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" podUID="3be6bca8-d8a9-426e-98fe-a7cbfde87f0f" containerName="payload-processing" containerID="cri-o://c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9" gracePeriod=25 Nov 03 10:19:41.558499 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:41.558437 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" podUID="7ba4cb4c-e6f0-4397-99fd-caa92d757b0d" containerName="payload-pre-processing" containerID="cri-o://e659342e3ddac2426c9c7bc5c5ff0d8d50dfa27cd354a6a4e393c5351d00b944" gracePeriod=25 Nov 03 10:19:46.902566 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:19:46.902527 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:19:47.911848 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:47.911812 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" podUID="3be6bca8-d8a9-426e-98fe-a7cbfde87f0f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:19:47.912329 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:47.911809 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" podUID="7ba4cb4c-e6f0-4397-99fd-caa92d757b0d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:19:56.913262 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:56.913226 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf"] Nov 03 10:19:56.913727 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:56.913445 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" containerID="cri-o://6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a" gracePeriod=30 Nov 03 10:19:57.911601 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:57.911558 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" podUID="7ba4cb4c-e6f0-4397-99fd-caa92d757b0d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:19:57.911796 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:57.911558 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" podUID="3be6bca8-d8a9-426e-98fe-a7cbfde87f0f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:19:58.491352 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.491316 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw"] Nov 03 10:19:58.540351 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.540314 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f6ckw"] Nov 03 10:19:58.615203 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.615166 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw"] Nov 03 10:19:58.615686 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.615671 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6d3afce2-7eb1-43c8-9d84-b912000d7063" containerName="revoke-keys" Nov 03 10:19:58.615760 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.615688 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3afce2-7eb1-43c8-9d84-b912000d7063" containerName="revoke-keys" Nov 03 10:19:58.615801 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.615792 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3afce2-7eb1-43c8-9d84-b912000d7063" containerName="revoke-keys" Nov 03 10:19:58.619313 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.619293 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:19:58.656577 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.656539 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xprdw\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:19:58.656795 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.656694 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xprdw\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:19:58.656795 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.656777 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5c2\" (UniqueName: \"kubernetes.io/projected/b214d719-7cd6-4101-8201-99a0528d2841-kube-api-access-zm5c2\") pod \"maas-api-revoke-sub-3d4156f1-xprdw\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:19:58.674162 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.674119 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:19:58.674360 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.674258 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:19:58.676361 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.676330 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw"] Nov 03 10:19:58.758370 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.758269 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5c2\" (UniqueName: \"kubernetes.io/projected/b214d719-7cd6-4101-8201-99a0528d2841-kube-api-access-zm5c2\") pod \"maas-api-revoke-sub-3d4156f1-xprdw\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:19:58.758579 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.758367 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xprdw\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:19:58.758579 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.758502 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xprdw\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:19:58.759353 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.759327 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xprdw\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:19:58.761112 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.761084 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xprdw\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:19:58.817709 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.817679 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5c2\" (UniqueName: \"kubernetes.io/projected/b214d719-7cd6-4101-8201-99a0528d2841-kube-api-access-zm5c2\") pod \"maas-api-revoke-sub-3d4156f1-xprdw\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:19:58.932552 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:58.932518 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:19:59.204971 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:59.204889 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0797dd-bcde-4c76-8e9f-7398f79aded4" path="/var/lib/kubelet/pods/5d0797dd-bcde-4c76-8e9f-7398f79aded4/volumes" Nov 03 10:19:59.313932 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:59.313904 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw"] Nov 03 10:19:59.315068 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:19:59.315043 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb214d719_7cd6_4101_8201_99a0528d2841.slice/crio-f01b04dfed815b443171d23381e65df41df1cbd1077bc714be8b7bd575c32f6c WatchSource:0}: Error finding container f01b04dfed815b443171d23381e65df41df1cbd1077bc714be8b7bd575c32f6c: Status 404 returned error can't find the container with id f01b04dfed815b443171d23381e65df41df1cbd1077bc714be8b7bd575c32f6c Nov 03 10:19:59.701532 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:59.701470 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" event={"ID":"b214d719-7cd6-4101-8201-99a0528d2841","Type":"ContainerStarted","Data":"bb6aae74513a9b8a46d1f1cb1d177c17fed2d7c194e15dfb5bce573a65af1531"} Nov 03 10:19:59.701976 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:59.701560 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" event={"ID":"b214d719-7cd6-4101-8201-99a0528d2841","Type":"ContainerStarted","Data":"f01b04dfed815b443171d23381e65df41df1cbd1077bc714be8b7bd575c32f6c"} Nov 03 10:19:59.819105 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:19:59.819004 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" podStartSLOduration=1.818990131 podStartE2EDuration="1.818990131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:19:59.816322023 +0000 UTC m=+6032.741264241" watchObservedRunningTime="2026-11-03 10:19:59.818990131 +0000 UTC m=+6032.743932339" Nov 03 10:20:00.709884 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:00.709794 5015 generic.go:356] "Generic (PLEG): container finished" podID="b214d719-7cd6-4101-8201-99a0528d2841" containerID="bb6aae74513a9b8a46d1f1cb1d177c17fed2d7c194e15dfb5bce573a65af1531" exitCode=0 Nov 03 10:20:00.709884 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:00.709831 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" event={"ID":"b214d719-7cd6-4101-8201-99a0528d2841","Type":"ContainerDied","Data":"bb6aae74513a9b8a46d1f1cb1d177c17fed2d7c194e15dfb5bce573a65af1531"} Nov 03 10:20:01.718238 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.718155 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:20:01.720448 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.720416 5015 generic.go:356] "Generic (PLEG): container finished" podID="7ba4cb4c-e6f0-4397-99fd-caa92d757b0d" containerID="e659342e3ddac2426c9c7bc5c5ff0d8d50dfa27cd354a6a4e393c5351d00b944" exitCode=0 Nov 03 10:20:01.720632 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.720489 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" event={"ID":"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d","Type":"ContainerDied","Data":"e659342e3ddac2426c9c7bc5c5ff0d8d50dfa27cd354a6a4e393c5351d00b944"} Nov 03 10:20:01.722180 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.722155 5015 generic.go:356] "Generic (PLEG): container finished" podID="3be6bca8-d8a9-426e-98fe-a7cbfde87f0f" containerID="c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9" exitCode=0 Nov 03 10:20:01.722326 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.722195 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" Nov 03 10:20:01.722326 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.722243 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" event={"ID":"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f","Type":"ContainerDied","Data":"c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9"} Nov 03 10:20:01.722326 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.722297 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk" event={"ID":"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f","Type":"ContainerDied","Data":"7c3e230021bc73189b44f16f9b68458d51ed050bcbc23a6023e9d5a26793c254"} Nov 03 10:20:01.722534 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.722331 5015 scope.go:122] "RemoveContainer" containerID="c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9" Nov 03 10:20:01.735518 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.735448 5015 scope.go:122] "RemoveContainer" containerID="c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9" Nov 03 10:20:01.735878 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:20:01.735847 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9\": container with ID starting with c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9 not found: ID does not exist" containerID="c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9" Nov 03 10:20:01.735994 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.735891 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9"} err="failed to get container status \"c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9\": rpc error: code = NotFound desc = could not find container \"c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9\": container with ID starting with c467b0b0eec2790ed76635a36a0dae8f74b1f994e328e4e70e5330f70e7d65f9 not found: ID does not exist" Nov 03 10:20:01.792982 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.792945 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-plugins-config-volume\") pod \"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f\" (UID: \"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f\") " Nov 03 10:20:01.793141 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.793062 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-kube-api-access-z9hhp\" (UniqueName: \"kubernetes.io/projected/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-kube-api-access-z9hhp\") pod \"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f\" (UID: \"3be6bca8-d8a9-426e-98fe-a7cbfde87f0f\") " Nov 03 10:20:01.793934 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.793902 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-plugins-config-volume" pod "3be6bca8-d8a9-426e-98fe-a7cbfde87f0f" (UID: "3be6bca8-d8a9-426e-98fe-a7cbfde87f0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:20:01.797836 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.797787 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-kube-api-access-z9hhp" pod "3be6bca8-d8a9-426e-98fe-a7cbfde87f0f" (UID: "3be6bca8-d8a9-426e-98fe-a7cbfde87f0f"). InnerVolumeSpecName "kube-api-access-z9hhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:20:01.840691 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.840659 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:20:01.893881 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.893856 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-plugins-config-volume\") pod \"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d\" (UID: \"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d\") " Nov 03 10:20:01.894048 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.894006 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-kube-api-access-mkntb\" (UniqueName: \"kubernetes.io/projected/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-kube-api-access-mkntb\") pod \"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d\" (UID: \"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d\") " Nov 03 10:20:01.894284 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.894247 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-plugins-config-volume" pod "7ba4cb4c-e6f0-4397-99fd-caa92d757b0d" (UID: "7ba4cb4c-e6f0-4397-99fd-caa92d757b0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:20:01.894433 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.894412 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:01.894522 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.894438 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9hhp\" (UniqueName: \"kubernetes.io/projected/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f-kube-api-access-z9hhp\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:01.894522 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.894454 5015 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-plugins-config-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:01.896270 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.896245 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-kube-api-access-mkntb" pod "7ba4cb4c-e6f0-4397-99fd-caa92d757b0d" (UID: "7ba4cb4c-e6f0-4397-99fd-caa92d757b0d"). InnerVolumeSpecName "kube-api-access-mkntb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:20:01.897651 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.897628 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:20:01.995160 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.995121 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-openshift-ca\") pod \"b214d719-7cd6-4101-8201-99a0528d2841\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " Nov 03 10:20:01.995371 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.995221 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b214d719-7cd6-4101-8201-99a0528d2841-kube-api-access-zm5c2\" (UniqueName: \"kubernetes.io/projected/b214d719-7cd6-4101-8201-99a0528d2841-kube-api-access-zm5c2\") pod \"b214d719-7cd6-4101-8201-99a0528d2841\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " Nov 03 10:20:01.995371 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.995326 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-xks-ca\") pod \"b214d719-7cd6-4101-8201-99a0528d2841\" (UID: \"b214d719-7cd6-4101-8201-99a0528d2841\") " Nov 03 10:20:01.995591 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.995576 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkntb\" (UniqueName: \"kubernetes.io/projected/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d-kube-api-access-mkntb\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:01.995668 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.995593 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-openshift-ca" pod "b214d719-7cd6-4101-8201-99a0528d2841" (UID: "b214d719-7cd6-4101-8201-99a0528d2841"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:20:01.997636 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.997607 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b214d719-7cd6-4101-8201-99a0528d2841-kube-api-access-zm5c2" pod "b214d719-7cd6-4101-8201-99a0528d2841" (UID: "b214d719-7cd6-4101-8201-99a0528d2841"). InnerVolumeSpecName "kube-api-access-zm5c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:20:01.997780 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:01.997640 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-xks-ca" pod "b214d719-7cd6-4101-8201-99a0528d2841" (UID: "b214d719-7cd6-4101-8201-99a0528d2841"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:20:02.054590 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.054523 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm"] Nov 03 10:20:02.060654 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.060612 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pdxlm"] Nov 03 10:20:02.096764 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.096730 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.096764 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.096762 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm5c2\" (UniqueName: \"kubernetes.io/projected/b214d719-7cd6-4101-8201-99a0528d2841-kube-api-access-zm5c2\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.096976 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.096781 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b214d719-7cd6-4101-8201-99a0528d2841-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.184361 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.184336 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:20:02.238543 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.238510 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk"] Nov 03 10:20:02.241325 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.241301 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65cb27-5c4b4ddb9pwwhk"] Nov 03 10:20:02.298624 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.298526 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istiod-ca-cert\") pod \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " Nov 03 10:20:02.298624 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.298580 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-credential-socket\") pod \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " Nov 03 10:20:02.298624 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.298613 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-envoy\") pod \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " Nov 03 10:20:02.298934 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.298651 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-certs\") pod \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " Nov 03 10:20:02.298934 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.298697 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-data\") pod \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " Nov 03 10:20:02.298934 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.298761 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-kube-api-access-t85hs\" (UniqueName: \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-kube-api-access-t85hs\") pod \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " Nov 03 10:20:02.298934 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.298814 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-token\" (UniqueName: \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-token\") pod \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " Nov 03 10:20:02.298934 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.298880 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-podinfo\") pod \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " Nov 03 10:20:02.298934 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.298887 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-credential-socket" pod "1eec8db9-80db-4ffa-acac-59dc5ad5d672" (UID: "1eec8db9-80db-4ffa-acac-59dc5ad5d672"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:20:02.298934 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.298926 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-socket\") pod \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\" (UID: \"1eec8db9-80db-4ffa-acac-59dc5ad5d672\") " Nov 03 10:20:02.299319 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.299026 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-certs" pod "1eec8db9-80db-4ffa-acac-59dc5ad5d672" (UID: "1eec8db9-80db-4ffa-acac-59dc5ad5d672"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:20:02.299319 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.299055 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istiod-ca-cert" pod "1eec8db9-80db-4ffa-acac-59dc5ad5d672" (UID: "1eec8db9-80db-4ffa-acac-59dc5ad5d672"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:20:02.299319 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.299281 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-socket" pod "1eec8db9-80db-4ffa-acac-59dc5ad5d672" (UID: "1eec8db9-80db-4ffa-acac-59dc5ad5d672"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:20:02.299514 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.299399 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-data" pod "1eec8db9-80db-4ffa-acac-59dc5ad5d672" (UID: "1eec8db9-80db-4ffa-acac-59dc5ad5d672"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:20:02.299717 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.299697 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.299817 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.299723 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.299817 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.299740 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.299944 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.299800 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.299944 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.299930 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.301328 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.301299 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-podinfo" pod "1eec8db9-80db-4ffa-acac-59dc5ad5d672" (UID: "1eec8db9-80db-4ffa-acac-59dc5ad5d672"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:20:02.301453 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.301425 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-kube-api-access-t85hs" pod "1eec8db9-80db-4ffa-acac-59dc5ad5d672" (UID: "1eec8db9-80db-4ffa-acac-59dc5ad5d672"). InnerVolumeSpecName "kube-api-access-t85hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:20:02.301757 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.301715 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-token" pod "1eec8db9-80db-4ffa-acac-59dc5ad5d672" (UID: "1eec8db9-80db-4ffa-acac-59dc5ad5d672"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:20:02.301757 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.301714 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-envoy" pod "1eec8db9-80db-4ffa-acac-59dc5ad5d672" (UID: "1eec8db9-80db-4ffa-acac-59dc5ad5d672"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:20:02.400993 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.400956 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t85hs\" (UniqueName: \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-kube-api-access-t85hs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.400993 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.400988 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.400993 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.401001 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.401247 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.401012 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eec8db9-80db-4ffa-acac-59dc5ad5d672-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:02.731416 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.731383 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" Nov 03 10:20:02.732018 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.731386 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr" event={"ID":"7ba4cb4c-e6f0-4397-99fd-caa92d757b0d","Type":"ContainerDied","Data":"f3c6bdfaba865441914252188b7ca6f232ec8cbbadf26dfee83d0747fccd29a1"} Nov 03 10:20:02.732018 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.731544 5015 scope.go:122] "RemoveContainer" containerID="e659342e3ddac2426c9c7bc5c5ff0d8d50dfa27cd354a6a4e393c5351d00b944" Nov 03 10:20:02.734443 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.734417 5015 generic.go:356] "Generic (PLEG): container finished" podID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerID="6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a" exitCode=0 Nov 03 10:20:02.734583 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.734509 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" Nov 03 10:20:02.734583 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.734508 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" event={"ID":"1eec8db9-80db-4ffa-acac-59dc5ad5d672","Type":"ContainerDied","Data":"6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a"} Nov 03 10:20:02.734583 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.734574 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf" event={"ID":"1eec8db9-80db-4ffa-acac-59dc5ad5d672","Type":"ContainerDied","Data":"d31197edfb8e791362901e60044606f672f7df4287ea7e90285ad664c4cb24be"} Nov 03 10:20:02.736141 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.736120 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" Nov 03 10:20:02.736280 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.736149 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw" event={"ID":"b214d719-7cd6-4101-8201-99a0528d2841","Type":"ContainerDied","Data":"f01b04dfed815b443171d23381e65df41df1cbd1077bc714be8b7bd575c32f6c"} Nov 03 10:20:02.736280 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.736195 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f01b04dfed815b443171d23381e65df41df1cbd1077bc714be8b7bd575c32f6c" Nov 03 10:20:02.744448 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.744428 5015 scope.go:122] "RemoveContainer" containerID="6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a" Nov 03 10:20:02.758353 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.758332 5015 scope.go:122] "RemoveContainer" containerID="cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d" Nov 03 10:20:02.767995 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.767967 5015 scope.go:122] "RemoveContainer" containerID="6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a" Nov 03 10:20:02.768273 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:20:02.768253 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a\": container with ID starting with 6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a not found: ID does not exist" containerID="6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a" Nov 03 10:20:02.768346 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.768283 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a"} err="failed to get container status \"6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a\": rpc error: code = NotFound desc = could not find container \"6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a\": container with ID starting with 6241cf1a167f3fefbe072dc99d8d0275c5a5356234b27872c66a8f60eba54d0a not found: ID does not exist" Nov 03 10:20:02.768346 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.768301 5015 scope.go:122] "RemoveContainer" containerID="cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d" Nov 03 10:20:02.768627 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:20:02.768594 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d\": container with ID starting with cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d not found: ID does not exist" containerID="cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d" Nov 03 10:20:02.768683 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.768633 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d"} err="failed to get container status \"cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d\": rpc error: code = NotFound desc = could not find container \"cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d\": container with ID starting with cd4ade6285172a5793580af0623f5933e40b76c5c3970c128bb21c6439a1ee2d not found: ID does not exist" Nov 03 10:20:02.908842 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.908809 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf"] Nov 03 10:20:02.914292 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:02.914263 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-65cb27-openshift-default-56b9d6b557sscwf"] Nov 03 10:20:03.027187 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:03.027098 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr"] Nov 03 10:20:03.029800 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:03.029779 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65cb27-76cf88bsqr"] Nov 03 10:20:03.203221 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:03.203184 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" path="/var/lib/kubelet/pods/1eec8db9-80db-4ffa-acac-59dc5ad5d672/volumes" Nov 03 10:20:03.203638 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:03.203622 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be6bca8-d8a9-426e-98fe-a7cbfde87f0f" path="/var/lib/kubelet/pods/3be6bca8-d8a9-426e-98fe-a7cbfde87f0f/volumes" Nov 03 10:20:03.204011 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:03.203998 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3ee15d-4073-44d1-b4db-2eb3edea6566" path="/var/lib/kubelet/pods/6b3ee15d-4073-44d1-b4db-2eb3edea6566/volumes" Nov 03 10:20:03.204690 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:03.204676 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba4cb4c-e6f0-4397-99fd-caa92d757b0d" path="/var/lib/kubelet/pods/7ba4cb4c-e6f0-4397-99fd-caa92d757b0d/volumes" Nov 03 10:20:16.902835 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:20:16.902795 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:20:17.550068 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.550034 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8"] Nov 03 10:20:17.550267 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.550234 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" podUID="05e4fc7a-c29d-4829-84f6-06cd99b8dae1" containerName="manager" containerID="cri-o://d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9" gracePeriod=10 Nov 03 10:20:17.816537 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.816443 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 10:20:17.832938 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.832904 5015 generic.go:356] "Generic (PLEG): container finished" podID="05e4fc7a-c29d-4829-84f6-06cd99b8dae1" containerID="d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9" exitCode=0 Nov 03 10:20:17.833091 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.832945 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" Nov 03 10:20:17.833091 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.832987 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" event={"ID":"05e4fc7a-c29d-4829-84f6-06cd99b8dae1","Type":"ContainerDied","Data":"d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9"} Nov 03 10:20:17.833091 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.833043 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8" event={"ID":"05e4fc7a-c29d-4829-84f6-06cd99b8dae1","Type":"ContainerDied","Data":"21107d4b528c620bd6959fcb096b29bd52496e23b5218d250c25d4931a914a69"} Nov 03 10:20:17.833222 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.833074 5015 scope.go:122] "RemoveContainer" containerID="d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9" Nov 03 10:20:17.847941 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.847918 5015 scope.go:122] "RemoveContainer" containerID="d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9" Nov 03 10:20:17.848260 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:20:17.848233 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9\": container with ID starting with d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9 not found: ID does not exist" containerID="d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9" Nov 03 10:20:17.848328 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.848271 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9"} err="failed to get container status \"d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9\": rpc error: code = NotFound desc = could not find container \"d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9\": container with ID starting with d62105d8cb0debf6f7754cc35b1786e456712dfbc14fc0d56c08359bf3a755c9 not found: ID does not exist" Nov 03 10:20:17.973594 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.973552 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-extensions-socket-volume\") pod \"05e4fc7a-c29d-4829-84f6-06cd99b8dae1\" (UID: \"05e4fc7a-c29d-4829-84f6-06cd99b8dae1\") " Nov 03 10:20:17.973993 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.973640 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-kube-api-access-j5j6h\" (UniqueName: \"kubernetes.io/projected/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-kube-api-access-j5j6h\") pod \"05e4fc7a-c29d-4829-84f6-06cd99b8dae1\" (UID: \"05e4fc7a-c29d-4829-84f6-06cd99b8dae1\") " Nov 03 10:20:17.973993 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.973981 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-extensions-socket-volume" pod "05e4fc7a-c29d-4829-84f6-06cd99b8dae1" (UID: "05e4fc7a-c29d-4829-84f6-06cd99b8dae1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:20:17.976051 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:17.976023 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-kube-api-access-j5j6h" pod "05e4fc7a-c29d-4829-84f6-06cd99b8dae1" (UID: "05e4fc7a-c29d-4829-84f6-06cd99b8dae1"). InnerVolumeSpecName "kube-api-access-j5j6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:20:18.074872 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:18.074786 5015 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-extensions-socket-volume\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:18.074872 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:18.074817 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5j6h\" (UniqueName: \"kubernetes.io/projected/05e4fc7a-c29d-4829-84f6-06cd99b8dae1-kube-api-access-j5j6h\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:20:18.258218 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:18.258185 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8"] Nov 03 10:20:18.263096 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:18.263060 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bqv8"] Nov 03 10:20:19.202902 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:19.202868 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e4fc7a-c29d-4829-84f6-06cd99b8dae1" path="/var/lib/kubelet/pods/05e4fc7a-c29d-4829-84f6-06cd99b8dae1/volumes" Nov 03 10:20:23.112563 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:23.112528 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6"] Nov 03 10:20:23.125851 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:23.125808 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvdv6"] Nov 03 10:20:23.203358 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:23.203320 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c72e97-f877-462d-877b-a706f2b4eb4c" path="/var/lib/kubelet/pods/67c72e97-f877-462d-877b-a706f2b4eb4c/volumes" Nov 03 10:20:30.036301 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:30.036258 5015 scope.go:122] "RemoveContainer" containerID="a5702acd79067d787af5e7b994860423e45046a633d566934e80b57ad2b20639" Nov 03 10:20:30.045886 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:30.045866 5015 scope.go:122] "RemoveContainer" containerID="d9351fc0ebe7a75d909923d6fd6d60960735228b3d4dc5787e2f55115ef0d3eb" Nov 03 10:20:46.902216 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:20:46.902124 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:20:54.054101 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:54.054063 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv"] Nov 03 10:20:54.103768 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:54.103731 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gqtjv"] Nov 03 10:20:55.203658 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:20:55.203619 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a608f782-0152-4296-ad6b-8ae268b6f4fb" path="/var/lib/kubelet/pods/a608f782-0152-4296-ad6b-8ae268b6f4fb/volumes" Nov 03 10:21:12.058143 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:12.058107 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr"] Nov 03 10:21:12.108521 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:12.108487 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-t55jr"] Nov 03 10:21:13.203356 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:13.203322 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0aed2a-9a79-4179-8ef0-059f1d5255ce" path="/var/lib/kubelet/pods/0e0aed2a-9a79-4179-8ef0-059f1d5255ce/volumes" Nov 03 10:21:16.900618 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:21:16.900582 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:21:21.195586 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.195549 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg"] Nov 03 10:21:21.196117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196019 5015 cpu_manager.go:414] "containerMap: removing container" podUID="3be6bca8-d8a9-426e-98fe-a7cbfde87f0f" containerName="payload-processing" Nov 03 10:21:21.196117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196032 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be6bca8-d8a9-426e-98fe-a7cbfde87f0f" containerName="payload-processing" Nov 03 10:21:21.196117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196057 5015 cpu_manager.go:414] "containerMap: removing container" podUID="05e4fc7a-c29d-4829-84f6-06cd99b8dae1" containerName="manager" Nov 03 10:21:21.196117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196062 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e4fc7a-c29d-4829-84f6-06cd99b8dae1" containerName="manager" Nov 03 10:21:21.196117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196072 5015 cpu_manager.go:414] "containerMap: removing container" podUID="7ba4cb4c-e6f0-4397-99fd-caa92d757b0d" containerName="payload-pre-processing" Nov 03 10:21:21.196117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196077 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba4cb4c-e6f0-4397-99fd-caa92d757b0d" containerName="payload-pre-processing" Nov 03 10:21:21.196117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196084 5015 cpu_manager.go:414] "containerMap: removing container" podUID="b214d719-7cd6-4101-8201-99a0528d2841" containerName="revoke-keys" Nov 03 10:21:21.196117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196089 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b214d719-7cd6-4101-8201-99a0528d2841" containerName="revoke-keys" Nov 03 10:21:21.196117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196099 5015 cpu_manager.go:414] "containerMap: removing container" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" Nov 03 10:21:21.196117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196104 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" Nov 03 10:21:21.196522 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196183 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e4fc7a-c29d-4829-84f6-06cd99b8dae1" containerName="manager" Nov 03 10:21:21.196522 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196193 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba4cb4c-e6f0-4397-99fd-caa92d757b0d" containerName="payload-pre-processing" Nov 03 10:21:21.196522 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196199 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="b214d719-7cd6-4101-8201-99a0528d2841" containerName="revoke-keys" Nov 03 10:21:21.196522 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196205 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" Nov 03 10:21:21.196522 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.196212 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be6bca8-d8a9-426e-98fe-a7cbfde87f0f" containerName="payload-processing" Nov 03 10:21:21.199506 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.199462 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" Nov 03 10:21:21.246921 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.246890 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg"] Nov 03 10:21:21.294196 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.294169 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-s4rff\"" Nov 03 10:21:21.306689 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.306659 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ph2\" (UniqueName: \"kubernetes.io/projected/19f7d098-9626-4724-a15e-85253b3dfd54-kube-api-access-n9ph2\") pod \"kuadrant-operator-controller-manager-678d87766f-cr2kg\" (UID: \"19f7d098-9626-4724-a15e-85253b3dfd54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" Nov 03 10:21:21.306831 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.306790 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19f7d098-9626-4724-a15e-85253b3dfd54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cr2kg\" (UID: \"19f7d098-9626-4724-a15e-85253b3dfd54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" Nov 03 10:21:21.407492 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.407439 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19f7d098-9626-4724-a15e-85253b3dfd54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cr2kg\" (UID: \"19f7d098-9626-4724-a15e-85253b3dfd54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" Nov 03 10:21:21.407647 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.407551 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ph2\" (UniqueName: \"kubernetes.io/projected/19f7d098-9626-4724-a15e-85253b3dfd54-kube-api-access-n9ph2\") pod \"kuadrant-operator-controller-manager-678d87766f-cr2kg\" (UID: \"19f7d098-9626-4724-a15e-85253b3dfd54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" Nov 03 10:21:21.407815 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.407797 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19f7d098-9626-4724-a15e-85253b3dfd54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cr2kg\" (UID: \"19f7d098-9626-4724-a15e-85253b3dfd54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" Nov 03 10:21:21.470679 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.470610 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ph2\" (UniqueName: \"kubernetes.io/projected/19f7d098-9626-4724-a15e-85253b3dfd54-kube-api-access-n9ph2\") pod \"kuadrant-operator-controller-manager-678d87766f-cr2kg\" (UID: \"19f7d098-9626-4724-a15e-85253b3dfd54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" Nov 03 10:21:21.514188 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.514164 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" Nov 03 10:21:21.994266 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:21.994235 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg"] Nov 03 10:21:21.995456 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:21:21.995430 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f7d098_9626_4724_a15e_85253b3dfd54.slice/crio-13700a5259a6551afa54a2a176f7b34ad2598978af06c25089c27221131fa565 WatchSource:0}: Error finding container 13700a5259a6551afa54a2a176f7b34ad2598978af06c25089c27221131fa565: Status 404 returned error can't find the container with id 13700a5259a6551afa54a2a176f7b34ad2598978af06c25089c27221131fa565 Nov 03 10:21:22.239958 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:22.239921 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" event={"ID":"19f7d098-9626-4724-a15e-85253b3dfd54","Type":"ContainerStarted","Data":"9c10ad919e3ce0da380c20e7f1a71007665462a923f140ff7dbb44cce31ae045"} Nov 03 10:21:22.240461 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:22.239980 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" Nov 03 10:21:22.240461 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:22.240003 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" event={"ID":"19f7d098-9626-4724-a15e-85253b3dfd54","Type":"ContainerStarted","Data":"13700a5259a6551afa54a2a176f7b34ad2598978af06c25089c27221131fa565"} Nov 03 10:21:22.360392 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:22.360290 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" podStartSLOduration=1.360274457 podStartE2EDuration="1.360274457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:21:22.358101991 +0000 UTC m=+6115.283044207" watchObservedRunningTime="2026-11-03 10:21:22.360274457 +0000 UTC m=+6115.285216673" Nov 03 10:21:30.093205 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:30.093175 5015 scope.go:122] "RemoveContainer" containerID="77a72afcd324f1884d7f1f47db79848e1cc52f04673bcda8dafc0220d00f7fb8" Nov 03 10:21:30.103036 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:30.103016 5015 scope.go:122] "RemoveContainer" containerID="b51ca39ef98795761398bc8fec50452ebafabb6ea200f9c5671b0cb84a8b8109" Nov 03 10:21:33.248098 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:33.248068 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cr2kg" Nov 03 10:21:45.512731 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:45.512664 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:21:45.513148 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:45.512788 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:21:46.902060 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:21:46.902016 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:21:47.028599 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.028567 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g"] Nov 03 10:21:47.029016 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.029004 5015 cpu_manager.go:414] "containerMap: removing container" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" Nov 03 10:21:47.029063 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.029018 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" Nov 03 10:21:47.029105 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.029100 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eec8db9-80db-4ffa-acac-59dc5ad5d672" containerName="istio-proxy" Nov 03 10:21:47.032352 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.032337 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:47.091468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.091433 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g"] Nov 03 10:21:47.117895 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.117861 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-x2g7g\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:47.118081 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.118000 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-x2g7g\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:47.118081 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.118033 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx2qf\" (UniqueName: \"kubernetes.io/projected/50f24bda-91fa-44fa-b39d-62087df957f9-kube-api-access-fx2qf\") pod \"maas-api-revoke-sub-d576e68b-x2g7g\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:47.137127 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.137095 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:21:47.137294 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.137162 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:21:47.219002 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.218909 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx2qf\" (UniqueName: \"kubernetes.io/projected/50f24bda-91fa-44fa-b39d-62087df957f9-kube-api-access-fx2qf\") pod \"maas-api-revoke-sub-d576e68b-x2g7g\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:47.219002 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.218974 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-x2g7g\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:47.219258 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.219041 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-x2g7g\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:47.219796 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.219728 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-x2g7g\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:47.221606 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.221585 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-x2g7g\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:47.278743 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.278710 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx2qf\" (UniqueName: \"kubernetes.io/projected/50f24bda-91fa-44fa-b39d-62087df957f9-kube-api-access-fx2qf\") pod \"maas-api-revoke-sub-d576e68b-x2g7g\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:47.345321 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.345291 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:47.725805 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:47.725780 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g"] Nov 03 10:21:47.727235 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:21:47.727195 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f24bda_91fa_44fa_b39d_62087df957f9.slice/crio-621c7a6e4440a79126612d526a1e71d00e07c7fbddc699136edcea7876cc6426 WatchSource:0}: Error finding container 621c7a6e4440a79126612d526a1e71d00e07c7fbddc699136edcea7876cc6426: Status 404 returned error can't find the container with id 621c7a6e4440a79126612d526a1e71d00e07c7fbddc699136edcea7876cc6426 Nov 03 10:21:48.433949 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:48.433860 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" event={"ID":"50f24bda-91fa-44fa-b39d-62087df957f9","Type":"ContainerStarted","Data":"7746b7590b27c2a7d0d3947a356c8f9e74bdfb4142183687a69bdbb6840de23e"} Nov 03 10:21:48.433949 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:48.433928 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" event={"ID":"50f24bda-91fa-44fa-b39d-62087df957f9","Type":"ContainerStarted","Data":"621c7a6e4440a79126612d526a1e71d00e07c7fbddc699136edcea7876cc6426"} Nov 03 10:21:48.550615 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:48.550560 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" podStartSLOduration=2.5505460859999998 podStartE2EDuration="2.550546086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:21:48.549586313 +0000 UTC m=+6141.474528554" watchObservedRunningTime="2026-11-03 10:21:48.550546086 +0000 UTC m=+6141.475488303" Nov 03 10:21:49.441457 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:49.441421 5015 generic.go:356] "Generic (PLEG): container finished" podID="50f24bda-91fa-44fa-b39d-62087df957f9" containerID="7746b7590b27c2a7d0d3947a356c8f9e74bdfb4142183687a69bdbb6840de23e" exitCode=0 Nov 03 10:21:49.441906 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:49.441518 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" event={"ID":"50f24bda-91fa-44fa-b39d-62087df957f9","Type":"ContainerDied","Data":"7746b7590b27c2a7d0d3947a356c8f9e74bdfb4142183687a69bdbb6840de23e"} Nov 03 10:21:50.597879 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:50.597854 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:50.655722 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:50.655689 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f24bda-91fa-44fa-b39d-62087df957f9-kube-api-access-fx2qf\" (UniqueName: \"kubernetes.io/projected/50f24bda-91fa-44fa-b39d-62087df957f9-kube-api-access-fx2qf\") pod \"50f24bda-91fa-44fa-b39d-62087df957f9\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " Nov 03 10:21:50.655859 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:50.655744 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-openshift-ca\") pod \"50f24bda-91fa-44fa-b39d-62087df957f9\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " Nov 03 10:21:50.655859 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:50.655816 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-xks-ca\") pod \"50f24bda-91fa-44fa-b39d-62087df957f9\" (UID: \"50f24bda-91fa-44fa-b39d-62087df957f9\") " Nov 03 10:21:50.656136 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:50.656109 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-openshift-ca" pod "50f24bda-91fa-44fa-b39d-62087df957f9" (UID: "50f24bda-91fa-44fa-b39d-62087df957f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:21:50.658102 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:50.658078 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f24bda-91fa-44fa-b39d-62087df957f9-kube-api-access-fx2qf" pod "50f24bda-91fa-44fa-b39d-62087df957f9" (UID: "50f24bda-91fa-44fa-b39d-62087df957f9"). InnerVolumeSpecName "kube-api-access-fx2qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:21:50.658275 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:50.658150 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-xks-ca" pod "50f24bda-91fa-44fa-b39d-62087df957f9" (UID: "50f24bda-91fa-44fa-b39d-62087df957f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:21:50.756854 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:50.756822 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:21:50.756854 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:50.756854 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx2qf\" (UniqueName: \"kubernetes.io/projected/50f24bda-91fa-44fa-b39d-62087df957f9-kube-api-access-fx2qf\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:21:50.757055 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:50.756865 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50f24bda-91fa-44fa-b39d-62087df957f9-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:21:51.454780 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:51.454753 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" Nov 03 10:21:51.454959 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:51.454755 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g" event={"ID":"50f24bda-91fa-44fa-b39d-62087df957f9","Type":"ContainerDied","Data":"621c7a6e4440a79126612d526a1e71d00e07c7fbddc699136edcea7876cc6426"} Nov 03 10:21:51.454959 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:21:51.454891 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="621c7a6e4440a79126612d526a1e71d00e07c7fbddc699136edcea7876cc6426" Nov 03 10:22:08.816708 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.816614 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg"] Nov 03 10:22:08.817317 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.817066 5015 cpu_manager.go:414] "containerMap: removing container" podUID="50f24bda-91fa-44fa-b39d-62087df957f9" containerName="revoke-keys" Nov 03 10:22:08.817317 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.817080 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f24bda-91fa-44fa-b39d-62087df957f9" containerName="revoke-keys" Nov 03 10:22:08.817317 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.817187 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f24bda-91fa-44fa-b39d-62087df957f9" containerName="revoke-keys" Nov 03 10:22:08.824034 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.824004 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:08.842682 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.842648 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d462n\" (UniqueName: \"kubernetes.io/projected/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-kube-api-access-d462n\") pod \"maas-api-revoke-sub-d2bc93bc-4t6bg\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:08.842837 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.842769 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4t6bg\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:08.842890 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.842846 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4t6bg\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:08.878574 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.878542 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg"] Nov 03 10:22:08.924669 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.924638 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:22:08.924843 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.924691 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:22:08.943771 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.943746 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d462n\" (UniqueName: \"kubernetes.io/projected/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-kube-api-access-d462n\") pod \"maas-api-revoke-sub-d2bc93bc-4t6bg\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:08.943912 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.943788 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4t6bg\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:08.943912 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.943830 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4t6bg\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:08.944679 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.944624 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4t6bg\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:08.946524 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:08.946470 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4t6bg\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:09.003504 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:09.003445 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d462n\" (UniqueName: \"kubernetes.io/projected/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-kube-api-access-d462n\") pod \"maas-api-revoke-sub-d2bc93bc-4t6bg\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:09.137393 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:09.137302 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:09.519752 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:09.519726 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg"] Nov 03 10:22:09.520644 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:22:09.520617 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a2fee7_c9e8_4fc5_a1c5_ac637b100c81.slice/crio-f4788a3cd3934fb56c9ebbcfaf06e57e961307ade3cbdaf04ab8127168e73738 WatchSource:0}: Error finding container f4788a3cd3934fb56c9ebbcfaf06e57e961307ade3cbdaf04ab8127168e73738: Status 404 returned error can't find the container with id f4788a3cd3934fb56c9ebbcfaf06e57e961307ade3cbdaf04ab8127168e73738 Nov 03 10:22:09.621568 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:09.621534 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" event={"ID":"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81","Type":"ContainerStarted","Data":"f4788a3cd3934fb56c9ebbcfaf06e57e961307ade3cbdaf04ab8127168e73738"} Nov 03 10:22:10.629218 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:10.629179 5015 generic.go:356] "Generic (PLEG): container finished" podID="b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81" containerID="819d587e6b98d9b95f15bd3eae8fb393f5020b13ade28e17cd69dd36971209bf" exitCode=0 Nov 03 10:22:10.629722 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:10.629232 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" event={"ID":"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81","Type":"ContainerDied","Data":"819d587e6b98d9b95f15bd3eae8fb393f5020b13ade28e17cd69dd36971209bf"} Nov 03 10:22:11.771412 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:11.771385 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:11.873623 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:11.873590 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-openshift-ca\") pod \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " Nov 03 10:22:11.873774 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:11.873697 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-kube-api-access-d462n\" (UniqueName: \"kubernetes.io/projected/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-kube-api-access-d462n\") pod \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " Nov 03 10:22:11.873774 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:11.873730 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-xks-ca\") pod \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\" (UID: \"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81\") " Nov 03 10:22:11.873973 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:11.873949 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-openshift-ca" pod "b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81" (UID: "b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:22:11.874059 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:11.874042 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:22:11.876044 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:11.876016 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-xks-ca" pod "b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81" (UID: "b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:22:11.876171 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:11.876067 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-kube-api-access-d462n" pod "b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81" (UID: "b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81"). InnerVolumeSpecName "kube-api-access-d462n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:22:11.974818 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:11.974788 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d462n\" (UniqueName: \"kubernetes.io/projected/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-kube-api-access-d462n\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:22:11.974818 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:11.974818 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:22:12.642292 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:12.642263 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" Nov 03 10:22:12.642506 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:12.642292 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg" event={"ID":"b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81","Type":"ContainerDied","Data":"f4788a3cd3934fb56c9ebbcfaf06e57e961307ade3cbdaf04ab8127168e73738"} Nov 03 10:22:12.642506 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:12.642334 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4788a3cd3934fb56c9ebbcfaf06e57e961307ade3cbdaf04ab8127168e73738" Nov 03 10:22:16.057774 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:16.057737 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn"] Nov 03 10:22:16.108213 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:16.108163 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sw2jn"] Nov 03 10:22:16.902230 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:22:16.902182 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:22:17.204943 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:17.204862 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="becc83f1-c9c5-40da-bac7-d529245cf0ac" path="/var/lib/kubelet/pods/becc83f1-c9c5-40da-bac7-d529245cf0ac/volumes" Nov 03 10:22:28.054775 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:28.054731 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h"] Nov 03 10:22:28.103410 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:28.103370 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-68z7h"] Nov 03 10:22:29.203220 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:29.203183 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc106d3-e936-4ffa-978e-f0e1c4b09e36" path="/var/lib/kubelet/pods/6fc106d3-e936-4ffa-978e-f0e1c4b09e36/volumes" Nov 03 10:22:30.132226 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:30.132195 5015 scope.go:122] "RemoveContainer" containerID="882c403704725c1f05b1cbfc9ee9b7b43c75fbf02fba3cd90a5d110ea2290312" Nov 03 10:22:30.142576 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:30.142554 5015 scope.go:122] "RemoveContainer" containerID="be02ffa89a21c76abc9ca36702379f6d7f791042dda2b318624e41c8b956753d" Nov 03 10:22:30.152523 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:30.152506 5015 scope.go:122] "RemoveContainer" containerID="226bcffa62adc569b1d3b3198e1630f29f47aa55152e37c4d6db0d4ef1b79d3c" Nov 03 10:22:45.506834 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:45.506776 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:22:45.507571 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:45.507524 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:22:45.793090 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:45.792572 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:22:45.793090 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:45.792636 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:22:46.901144 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:22:46.901103 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:22:49.816190 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.816153 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk"] Nov 03 10:22:49.816668 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.816645 5015 cpu_manager.go:414] "containerMap: removing container" podUID="b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81" containerName="revoke-keys" Nov 03 10:22:49.816668 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.816658 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81" containerName="revoke-keys" Nov 03 10:22:49.816776 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.816764 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81" containerName="revoke-keys" Nov 03 10:22:49.820054 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.820038 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:49.836968 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.836946 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb20ca37-pxdwk\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:49.837096 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.836993 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb20ca37-pxdwk\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:49.837096 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.837084 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpbct\" (UniqueName: \"kubernetes.io/projected/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-kube-api-access-kpbct\") pod \"maas-api-revoke-sub-eb20ca37-pxdwk\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:49.877956 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.877931 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk"] Nov 03 10:22:49.924111 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.924073 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:22:49.924269 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.924134 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:22:49.937880 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.937854 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb20ca37-pxdwk\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:49.938029 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.937917 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb20ca37-pxdwk\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:49.938029 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.937947 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbct\" (UniqueName: \"kubernetes.io/projected/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-kube-api-access-kpbct\") pod \"maas-api-revoke-sub-eb20ca37-pxdwk\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:49.938714 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.938684 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb20ca37-pxdwk\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:49.940485 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.940461 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb20ca37-pxdwk\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:49.997076 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:49.997052 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbct\" (UniqueName: \"kubernetes.io/projected/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-kube-api-access-kpbct\") pod \"maas-api-revoke-sub-eb20ca37-pxdwk\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:50.132750 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:50.132678 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:50.510002 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:50.509973 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk"] Nov 03 10:22:50.511346 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:22:50.511318 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8a4e0f_f857_4d07_9bc0_c4a39ac79df2.slice/crio-0444d7f2d8401a05131e3977ecac8768f690a84f2b87429440f73c105746d23b WatchSource:0}: Error finding container 0444d7f2d8401a05131e3977ecac8768f690a84f2b87429440f73c105746d23b: Status 404 returned error can't find the container with id 0444d7f2d8401a05131e3977ecac8768f690a84f2b87429440f73c105746d23b Nov 03 10:22:50.931927 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:50.931893 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" event={"ID":"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2","Type":"ContainerStarted","Data":"4756c39073983164f3b4a3653abba800b8667089e450759824c3ac71fa01e410"} Nov 03 10:22:50.932316 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:50.931943 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" event={"ID":"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2","Type":"ContainerStarted","Data":"0444d7f2d8401a05131e3977ecac8768f690a84f2b87429440f73c105746d23b"} Nov 03 10:22:51.050740 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:51.050689 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" podStartSLOduration=2.050674953 podStartE2EDuration="2.050674953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:22:51.048273263 +0000 UTC m=+6203.973215479" watchObservedRunningTime="2026-11-03 10:22:51.050674953 +0000 UTC m=+6203.975617168" Nov 03 10:22:51.939328 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:51.939293 5015 generic.go:356] "Generic (PLEG): container finished" podID="3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2" containerID="4756c39073983164f3b4a3653abba800b8667089e450759824c3ac71fa01e410" exitCode=0 Nov 03 10:22:51.939815 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:51.939383 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" event={"ID":"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2","Type":"ContainerDied","Data":"4756c39073983164f3b4a3653abba800b8667089e450759824c3ac71fa01e410"} Nov 03 10:22:53.100049 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.100026 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:53.167128 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.167101 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-xks-ca\") pod \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " Nov 03 10:22:53.167280 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.167155 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-kube-api-access-kpbct\" (UniqueName: \"kubernetes.io/projected/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-kube-api-access-kpbct\") pod \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " Nov 03 10:22:53.167280 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.167235 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-openshift-ca\") pod \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\" (UID: \"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2\") " Nov 03 10:22:53.167652 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.167628 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-openshift-ca" pod "3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2" (UID: "3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:22:53.169559 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.169510 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-kube-api-access-kpbct" pod "3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2" (UID: "3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2"). InnerVolumeSpecName "kube-api-access-kpbct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:22:53.169559 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.169521 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-xks-ca" pod "3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2" (UID: "3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:22:53.268463 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.268431 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:22:53.268463 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.268460 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:22:53.268463 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.268469 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpbct\" (UniqueName: \"kubernetes.io/projected/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2-kube-api-access-kpbct\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:22:53.954511 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.954456 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" Nov 03 10:22:53.954511 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.954502 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk" event={"ID":"3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2","Type":"ContainerDied","Data":"0444d7f2d8401a05131e3977ecac8768f690a84f2b87429440f73c105746d23b"} Nov 03 10:22:53.954742 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:22:53.954576 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0444d7f2d8401a05131e3977ecac8768f690a84f2b87429440f73c105746d23b" Nov 03 10:23:00.576873 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.576832 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq"] Nov 03 10:23:00.577591 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.577568 5015 cpu_manager.go:414] "containerMap: removing container" podUID="3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2" containerName="revoke-keys" Nov 03 10:23:00.577654 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.577597 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2" containerName="revoke-keys" Nov 03 10:23:00.577807 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.577792 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2" containerName="revoke-keys" Nov 03 10:23:00.581855 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.581836 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:00.638771 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.638738 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq"] Nov 03 10:23:00.638771 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.638764 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whmc\" (UniqueName: \"kubernetes.io/projected/8ee675d0-0851-4019-b55f-124f30eb3c7a-kube-api-access-4whmc\") pod \"maas-api-revoke-sub-770a93b5-zj8jq\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:00.639012 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.638794 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-770a93b5-zj8jq\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:00.639012 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.638961 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-770a93b5-zj8jq\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:00.686012 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.685982 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:23:00.686012 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.685983 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:23:00.739503 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.739461 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whmc\" (UniqueName: \"kubernetes.io/projected/8ee675d0-0851-4019-b55f-124f30eb3c7a-kube-api-access-4whmc\") pod \"maas-api-revoke-sub-770a93b5-zj8jq\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:00.739657 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.739510 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-770a93b5-zj8jq\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:00.739657 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.739571 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-770a93b5-zj8jq\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:00.740296 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.740215 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-770a93b5-zj8jq\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:00.741883 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.741864 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-770a93b5-zj8jq\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:00.799037 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.799007 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whmc\" (UniqueName: \"kubernetes.io/projected/8ee675d0-0851-4019-b55f-124f30eb3c7a-kube-api-access-4whmc\") pod \"maas-api-revoke-sub-770a93b5-zj8jq\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:00.894906 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:00.894835 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:01.272286 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:01.272259 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq"] Nov 03 10:23:01.273074 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:23:01.273047 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee675d0_0851_4019_b55f_124f30eb3c7a.slice/crio-d5985a0e3758ed30e0f773e9b63d02fe1fe67eec9f2f63a2b24f6712cdb8eae5 WatchSource:0}: Error finding container d5985a0e3758ed30e0f773e9b63d02fe1fe67eec9f2f63a2b24f6712cdb8eae5: Status 404 returned error can't find the container with id d5985a0e3758ed30e0f773e9b63d02fe1fe67eec9f2f63a2b24f6712cdb8eae5 Nov 03 10:23:02.008288 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:02.008253 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" event={"ID":"8ee675d0-0851-4019-b55f-124f30eb3c7a","Type":"ContainerStarted","Data":"4f11233e5b461fcf00a5dd21f2677c54977a92211e8f6a74a1b7d8da1c3fff18"} Nov 03 10:23:02.008707 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:02.008306 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" event={"ID":"8ee675d0-0851-4019-b55f-124f30eb3c7a","Type":"ContainerStarted","Data":"d5985a0e3758ed30e0f773e9b63d02fe1fe67eec9f2f63a2b24f6712cdb8eae5"} Nov 03 10:23:02.127292 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:02.127242 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" podStartSLOduration=2.127228031 podStartE2EDuration="2.127228031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:23:02.124587683 +0000 UTC m=+6215.049529920" watchObservedRunningTime="2026-11-03 10:23:02.127228031 +0000 UTC m=+6215.052170300" Nov 03 10:23:03.015862 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:03.015826 5015 generic.go:356] "Generic (PLEG): container finished" podID="8ee675d0-0851-4019-b55f-124f30eb3c7a" containerID="4f11233e5b461fcf00a5dd21f2677c54977a92211e8f6a74a1b7d8da1c3fff18" exitCode=0 Nov 03 10:23:03.016319 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:03.015915 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" event={"ID":"8ee675d0-0851-4019-b55f-124f30eb3c7a","Type":"ContainerDied","Data":"4f11233e5b461fcf00a5dd21f2677c54977a92211e8f6a74a1b7d8da1c3fff18"} Nov 03 10:23:04.173848 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:04.173821 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:04.276725 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:04.276633 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-xks-ca\") pod \"8ee675d0-0851-4019-b55f-124f30eb3c7a\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " Nov 03 10:23:04.276725 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:04.276704 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-openshift-ca\") pod \"8ee675d0-0851-4019-b55f-124f30eb3c7a\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " Nov 03 10:23:04.277019 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:04.276769 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee675d0-0851-4019-b55f-124f30eb3c7a-kube-api-access-4whmc\" (UniqueName: \"kubernetes.io/projected/8ee675d0-0851-4019-b55f-124f30eb3c7a-kube-api-access-4whmc\") pod \"8ee675d0-0851-4019-b55f-124f30eb3c7a\" (UID: \"8ee675d0-0851-4019-b55f-124f30eb3c7a\") " Nov 03 10:23:04.277092 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:04.277017 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-openshift-ca" pod "8ee675d0-0851-4019-b55f-124f30eb3c7a" (UID: "8ee675d0-0851-4019-b55f-124f30eb3c7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:23:04.277170 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:04.277147 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:23:04.279121 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:04.279091 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-xks-ca" pod "8ee675d0-0851-4019-b55f-124f30eb3c7a" (UID: "8ee675d0-0851-4019-b55f-124f30eb3c7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:23:04.279308 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:04.279288 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee675d0-0851-4019-b55f-124f30eb3c7a-kube-api-access-4whmc" pod "8ee675d0-0851-4019-b55f-124f30eb3c7a" (UID: "8ee675d0-0851-4019-b55f-124f30eb3c7a"). InnerVolumeSpecName "kube-api-access-4whmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:23:04.378095 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:04.378057 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whmc\" (UniqueName: \"kubernetes.io/projected/8ee675d0-0851-4019-b55f-124f30eb3c7a-kube-api-access-4whmc\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:23:04.378095 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:04.378091 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee675d0-0851-4019-b55f-124f30eb3c7a-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:23:05.028849 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:05.028813 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" event={"ID":"8ee675d0-0851-4019-b55f-124f30eb3c7a","Type":"ContainerDied","Data":"d5985a0e3758ed30e0f773e9b63d02fe1fe67eec9f2f63a2b24f6712cdb8eae5"} Nov 03 10:23:05.028849 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:05.028848 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq" Nov 03 10:23:05.029081 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:05.028873 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5985a0e3758ed30e0f773e9b63d02fe1fe67eec9f2f63a2b24f6712cdb8eae5" Nov 03 10:23:16.900641 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:23:16.900598 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:23:30.187266 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:30.187234 5015 scope.go:122] "RemoveContainer" containerID="8f212e7aaa6ef1bd9499d84230e74e0873fcf05cfaf4018037dffb1b2fadd248" Nov 03 10:23:30.197123 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:30.197103 5015 scope.go:122] "RemoveContainer" containerID="9a7483bcb9ffc63b7d992af7a911367962c5f564325115fba997ccd4809fadb0" Nov 03 10:23:30.206176 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:30.206160 5015 scope.go:122] "RemoveContainer" containerID="231f1e493095d949e56cdd5d21857af1a639330ccb75752ea6a1dd08757364d5" Nov 03 10:23:36.091983 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.091891 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf"] Nov 03 10:23:36.092387 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.092328 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8ee675d0-0851-4019-b55f-124f30eb3c7a" containerName="revoke-keys" Nov 03 10:23:36.092387 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.092338 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee675d0-0851-4019-b55f-124f30eb3c7a" containerName="revoke-keys" Nov 03 10:23:36.092491 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.092416 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee675d0-0851-4019-b55f-124f30eb3c7a" containerName="revoke-keys" Nov 03 10:23:36.096922 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.096899 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:36.145743 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.145706 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5be9a74-fcdpf\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:36.145959 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.145774 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5be9a74-fcdpf\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:36.145959 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.145881 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljspr\" (UniqueName: \"kubernetes.io/projected/9204425e-8e3c-4f97-8670-5d6532a03204-kube-api-access-ljspr\") pod \"maas-api-revoke-sub-a5be9a74-fcdpf\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:36.151945 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.151919 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf"] Nov 03 10:23:36.198737 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.198702 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:23:36.198737 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.198717 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:23:36.246876 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.246842 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5be9a74-fcdpf\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:36.247156 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.246933 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5be9a74-fcdpf\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:36.247156 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.246991 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljspr\" (UniqueName: \"kubernetes.io/projected/9204425e-8e3c-4f97-8670-5d6532a03204-kube-api-access-ljspr\") pod \"maas-api-revoke-sub-a5be9a74-fcdpf\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:36.247534 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.247512 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5be9a74-fcdpf\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:36.249421 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.249399 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5be9a74-fcdpf\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:36.307835 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.307799 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljspr\" (UniqueName: \"kubernetes.io/projected/9204425e-8e3c-4f97-8670-5d6532a03204-kube-api-access-ljspr\") pod \"maas-api-revoke-sub-a5be9a74-fcdpf\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:36.409943 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.409849 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:36.787218 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:36.787189 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf"] Nov 03 10:23:36.788295 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:23:36.788268 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9204425e_8e3c_4f97_8670_5d6532a03204.slice/crio-fb18fdf65e3bc9dee2158eee195be24b946a984ca537167098889d5f3519dc16 WatchSource:0}: Error finding container fb18fdf65e3bc9dee2158eee195be24b946a984ca537167098889d5f3519dc16: Status 404 returned error can't find the container with id fb18fdf65e3bc9dee2158eee195be24b946a984ca537167098889d5f3519dc16 Nov 03 10:23:37.255652 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:37.255613 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" event={"ID":"9204425e-8e3c-4f97-8670-5d6532a03204","Type":"ContainerStarted","Data":"cc5ab99358d2286ff7ca776c6facc540f9c33ccbd3509fb7b788ad99bb6b7d4c"} Nov 03 10:23:37.256037 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:37.255676 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" event={"ID":"9204425e-8e3c-4f97-8670-5d6532a03204","Type":"ContainerStarted","Data":"fb18fdf65e3bc9dee2158eee195be24b946a984ca537167098889d5f3519dc16"} Nov 03 10:23:37.374117 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:37.374009 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" podStartSLOduration=1.373993579 podStartE2EDuration="1.373993579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:23:37.371614423 +0000 UTC m=+6250.296556657" watchObservedRunningTime="2026-11-03 10:23:37.373993579 +0000 UTC m=+6250.298935795" Nov 03 10:23:38.263746 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:38.263703 5015 generic.go:356] "Generic (PLEG): container finished" podID="9204425e-8e3c-4f97-8670-5d6532a03204" containerID="cc5ab99358d2286ff7ca776c6facc540f9c33ccbd3509fb7b788ad99bb6b7d4c" exitCode=0 Nov 03 10:23:38.264147 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:38.263795 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" event={"ID":"9204425e-8e3c-4f97-8670-5d6532a03204","Type":"ContainerDied","Data":"cc5ab99358d2286ff7ca776c6facc540f9c33ccbd3509fb7b788ad99bb6b7d4c"} Nov 03 10:23:39.410022 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:39.409997 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:39.478629 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:39.478592 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-xks-ca\") pod \"9204425e-8e3c-4f97-8670-5d6532a03204\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " Nov 03 10:23:39.478787 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:39.478655 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-openshift-ca\") pod \"9204425e-8e3c-4f97-8670-5d6532a03204\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " Nov 03 10:23:39.478787 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:39.478700 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9204425e-8e3c-4f97-8670-5d6532a03204-kube-api-access-ljspr\" (UniqueName: \"kubernetes.io/projected/9204425e-8e3c-4f97-8670-5d6532a03204-kube-api-access-ljspr\") pod \"9204425e-8e3c-4f97-8670-5d6532a03204\" (UID: \"9204425e-8e3c-4f97-8670-5d6532a03204\") " Nov 03 10:23:39.478995 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:39.478973 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-openshift-ca" pod "9204425e-8e3c-4f97-8670-5d6532a03204" (UID: "9204425e-8e3c-4f97-8670-5d6532a03204"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:23:39.480928 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:39.480903 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-xks-ca" pod "9204425e-8e3c-4f97-8670-5d6532a03204" (UID: "9204425e-8e3c-4f97-8670-5d6532a03204"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:23:39.481005 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:39.480990 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9204425e-8e3c-4f97-8670-5d6532a03204-kube-api-access-ljspr" pod "9204425e-8e3c-4f97-8670-5d6532a03204" (UID: "9204425e-8e3c-4f97-8670-5d6532a03204"). InnerVolumeSpecName "kube-api-access-ljspr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:23:39.579658 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:39.579584 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:23:39.579658 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:39.579610 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9204425e-8e3c-4f97-8670-5d6532a03204-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:23:39.579658 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:39.579621 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljspr\" (UniqueName: \"kubernetes.io/projected/9204425e-8e3c-4f97-8670-5d6532a03204-kube-api-access-ljspr\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:23:40.276849 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:40.276818 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" Nov 03 10:23:40.277034 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:40.276862 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf" event={"ID":"9204425e-8e3c-4f97-8670-5d6532a03204","Type":"ContainerDied","Data":"fb18fdf65e3bc9dee2158eee195be24b946a984ca537167098889d5f3519dc16"} Nov 03 10:23:40.277034 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:23:40.276924 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb18fdf65e3bc9dee2158eee195be24b946a984ca537167098889d5f3519dc16" Nov 03 10:23:46.902296 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:23:46.902260 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:24:16.908954 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:24:16.908914 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:24:19.055695 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:19.055605 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs"] Nov 03 10:24:19.105342 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:19.105311 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b553a7a9-88qxs"] Nov 03 10:24:19.203360 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:19.203317 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3afce2-7eb1-43c8-9d84-b912000d7063" path="/var/lib/kubelet/pods/6d3afce2-7eb1-43c8-9d84-b912000d7063/volumes" Nov 03 10:24:20.810334 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:20.810295 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d"] Nov 03 10:24:20.810891 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:20.810804 5015 cpu_manager.go:414] "containerMap: removing container" podUID="9204425e-8e3c-4f97-8670-5d6532a03204" containerName="revoke-keys" Nov 03 10:24:20.810891 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:20.810817 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9204425e-8e3c-4f97-8670-5d6532a03204" containerName="revoke-keys" Nov 03 10:24:20.810891 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:20.810886 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="9204425e-8e3c-4f97-8670-5d6532a03204" containerName="revoke-keys" Nov 03 10:24:20.814073 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:20.814051 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:20.871052 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:20.871022 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d"] Nov 03 10:24:20.911231 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:20.911203 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/968346cd-8243-4b92-94da-bd610905fad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587ba938-sgt9d\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:20.911383 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:20.911265 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vpnp\" (UniqueName: \"kubernetes.io/projected/968346cd-8243-4b92-94da-bd610905fad5-kube-api-access-5vpnp\") pod \"maas-api-revoke-sub-587ba938-sgt9d\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:20.911383 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:20.911373 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/968346cd-8243-4b92-94da-bd610905fad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587ba938-sgt9d\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:20.917739 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:20.917711 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:24:20.917833 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:20.917748 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:24:21.012502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:21.012440 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/968346cd-8243-4b92-94da-bd610905fad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587ba938-sgt9d\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:21.012671 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:21.012555 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vpnp\" (UniqueName: \"kubernetes.io/projected/968346cd-8243-4b92-94da-bd610905fad5-kube-api-access-5vpnp\") pod \"maas-api-revoke-sub-587ba938-sgt9d\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:21.012671 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:21.012624 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/968346cd-8243-4b92-94da-bd610905fad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587ba938-sgt9d\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:21.013174 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:21.013146 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/968346cd-8243-4b92-94da-bd610905fad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587ba938-sgt9d\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:21.015177 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:21.015158 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/968346cd-8243-4b92-94da-bd610905fad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587ba938-sgt9d\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:21.071617 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:21.071558 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vpnp\" (UniqueName: \"kubernetes.io/projected/968346cd-8243-4b92-94da-bd610905fad5-kube-api-access-5vpnp\") pod \"maas-api-revoke-sub-587ba938-sgt9d\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:21.127246 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:21.127221 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:21.505696 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:21.505671 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d"] Nov 03 10:24:21.507105 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:24:21.507080 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968346cd_8243_4b92_94da_bd610905fad5.slice/crio-4f3c9649e913d82ecd3bd2046cd0b017a1a96e7bdb4b528f003afe8ce540a5e9 WatchSource:0}: Error finding container 4f3c9649e913d82ecd3bd2046cd0b017a1a96e7bdb4b528f003afe8ce540a5e9: Status 404 returned error can't find the container with id 4f3c9649e913d82ecd3bd2046cd0b017a1a96e7bdb4b528f003afe8ce540a5e9 Nov 03 10:24:21.559224 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:21.559194 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" event={"ID":"968346cd-8243-4b92-94da-bd610905fad5","Type":"ContainerStarted","Data":"4f3c9649e913d82ecd3bd2046cd0b017a1a96e7bdb4b528f003afe8ce540a5e9"} Nov 03 10:24:22.567628 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:22.567595 5015 generic.go:356] "Generic (PLEG): container finished" podID="968346cd-8243-4b92-94da-bd610905fad5" containerID="da7800a9ade9d0fea0fda4f5d51f9d03373582be8108772da51acda61c303265" exitCode=0 Nov 03 10:24:22.568017 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:22.567674 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" event={"ID":"968346cd-8243-4b92-94da-bd610905fad5","Type":"ContainerDied","Data":"da7800a9ade9d0fea0fda4f5d51f9d03373582be8108772da51acda61c303265"} Nov 03 10:24:23.712489 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:23.712456 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:23.841696 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:23.841622 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/968346cd-8243-4b92-94da-bd610905fad5-kube-api-access-5vpnp\" (UniqueName: \"kubernetes.io/projected/968346cd-8243-4b92-94da-bd610905fad5-kube-api-access-5vpnp\") pod \"968346cd-8243-4b92-94da-bd610905fad5\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " Nov 03 10:24:23.841696 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:23.841672 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/968346cd-8243-4b92-94da-bd610905fad5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/968346cd-8243-4b92-94da-bd610905fad5-maas-api-openshift-ca\") pod \"968346cd-8243-4b92-94da-bd610905fad5\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " Nov 03 10:24:23.841936 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:23.841791 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/968346cd-8243-4b92-94da-bd610905fad5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/968346cd-8243-4b92-94da-bd610905fad5-maas-api-xks-ca\") pod \"968346cd-8243-4b92-94da-bd610905fad5\" (UID: \"968346cd-8243-4b92-94da-bd610905fad5\") " Nov 03 10:24:23.842147 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:23.842113 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968346cd-8243-4b92-94da-bd610905fad5-maas-api-openshift-ca" pod "968346cd-8243-4b92-94da-bd610905fad5" (UID: "968346cd-8243-4b92-94da-bd610905fad5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:24:23.843975 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:23.843952 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/968346cd-8243-4b92-94da-bd610905fad5-maas-api-xks-ca" pod "968346cd-8243-4b92-94da-bd610905fad5" (UID: "968346cd-8243-4b92-94da-bd610905fad5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:24:23.844543 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:23.844510 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968346cd-8243-4b92-94da-bd610905fad5-kube-api-access-5vpnp" pod "968346cd-8243-4b92-94da-bd610905fad5" (UID: "968346cd-8243-4b92-94da-bd610905fad5"). InnerVolumeSpecName "kube-api-access-5vpnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:24:23.943418 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:23.943388 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/968346cd-8243-4b92-94da-bd610905fad5-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:24:23.943418 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:23.943415 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vpnp\" (UniqueName: \"kubernetes.io/projected/968346cd-8243-4b92-94da-bd610905fad5-kube-api-access-5vpnp\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:24:23.943618 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:23.943425 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/968346cd-8243-4b92-94da-bd610905fad5-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:24:24.581720 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:24.581691 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" Nov 03 10:24:24.581911 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:24.581690 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d" event={"ID":"968346cd-8243-4b92-94da-bd610905fad5","Type":"ContainerDied","Data":"4f3c9649e913d82ecd3bd2046cd0b017a1a96e7bdb4b528f003afe8ce540a5e9"} Nov 03 10:24:24.581911 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:24.581817 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f3c9649e913d82ecd3bd2046cd0b017a1a96e7bdb4b528f003afe8ce540a5e9" Nov 03 10:24:28.361450 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:28.361420 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:24:28.387060 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:28.387037 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:24:30.236885 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:30.236849 5015 scope.go:122] "RemoveContainer" containerID="0fde7aa10420d6abaa59ab314a800a5a4e487a58d2772f158813822350577bb9" Nov 03 10:24:31.611182 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.611102 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7"] Nov 03 10:24:31.611768 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.611749 5015 cpu_manager.go:414] "containerMap: removing container" podUID="968346cd-8243-4b92-94da-bd610905fad5" containerName="revoke-keys" Nov 03 10:24:31.611823 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.611772 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="968346cd-8243-4b92-94da-bd610905fad5" containerName="revoke-keys" Nov 03 10:24:31.611921 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.611909 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="968346cd-8243-4b92-94da-bd610905fad5" containerName="revoke-keys" Nov 03 10:24:31.616729 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.616711 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:31.670461 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.670427 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7"] Nov 03 10:24:31.714684 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.714650 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d5f895-l5dd7\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:31.714875 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.714695 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2mnv\" (UniqueName: \"kubernetes.io/projected/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-kube-api-access-d2mnv\") pod \"maas-api-revoke-sub-a3d5f895-l5dd7\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:31.714875 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.714773 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d5f895-l5dd7\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:31.717468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.717445 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:24:31.717468 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.717446 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:24:31.815293 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.815261 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d5f895-l5dd7\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:31.815508 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.815304 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2mnv\" (UniqueName: \"kubernetes.io/projected/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-kube-api-access-d2mnv\") pod \"maas-api-revoke-sub-a3d5f895-l5dd7\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:31.815508 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.815426 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d5f895-l5dd7\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:31.816143 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.816048 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d5f895-l5dd7\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:31.818122 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.818097 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d5f895-l5dd7\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:31.874929 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.874862 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2mnv\" (UniqueName: \"kubernetes.io/projected/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-kube-api-access-d2mnv\") pod \"maas-api-revoke-sub-a3d5f895-l5dd7\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:31.930842 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:31.930811 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:32.307217 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:32.307185 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7"] Nov 03 10:24:32.308464 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:24:32.308428 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95d0313_2000_4c56_a7fa_d4d4a30e23cb.slice/crio-91c093206df57c6d8ed7b5a19207dd55abff736098d01164053157431f42a422 WatchSource:0}: Error finding container 91c093206df57c6d8ed7b5a19207dd55abff736098d01164053157431f42a422: Status 404 returned error can't find the container with id 91c093206df57c6d8ed7b5a19207dd55abff736098d01164053157431f42a422 Nov 03 10:24:32.633079 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:32.632998 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" event={"ID":"a95d0313-2000-4c56-a7fa-d4d4a30e23cb","Type":"ContainerStarted","Data":"91c093206df57c6d8ed7b5a19207dd55abff736098d01164053157431f42a422"} Nov 03 10:24:33.640062 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:33.640027 5015 generic.go:356] "Generic (PLEG): container finished" podID="a95d0313-2000-4c56-a7fa-d4d4a30e23cb" containerID="924587d3163b61b7d4be4c4065d15dbb5416be5acc1835ad55466eb5e8275841" exitCode=0 Nov 03 10:24:33.640572 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:33.640111 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" event={"ID":"a95d0313-2000-4c56-a7fa-d4d4a30e23cb","Type":"ContainerDied","Data":"924587d3163b61b7d4be4c4065d15dbb5416be5acc1835ad55466eb5e8275841"} Nov 03 10:24:34.789635 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:34.789611 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:34.843409 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:34.843378 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-openshift-ca\") pod \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " Nov 03 10:24:34.843594 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:34.843506 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-kube-api-access-d2mnv\" (UniqueName: \"kubernetes.io/projected/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-kube-api-access-d2mnv\") pod \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " Nov 03 10:24:34.843594 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:34.843575 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-xks-ca\") pod \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\" (UID: \"a95d0313-2000-4c56-a7fa-d4d4a30e23cb\") " Nov 03 10:24:34.843769 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:34.843745 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-openshift-ca" pod "a95d0313-2000-4c56-a7fa-d4d4a30e23cb" (UID: "a95d0313-2000-4c56-a7fa-d4d4a30e23cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:24:34.843933 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:34.843917 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:24:34.845844 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:34.845823 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-xks-ca" pod "a95d0313-2000-4c56-a7fa-d4d4a30e23cb" (UID: "a95d0313-2000-4c56-a7fa-d4d4a30e23cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:24:34.845931 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:34.845845 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-kube-api-access-d2mnv" pod "a95d0313-2000-4c56-a7fa-d4d4a30e23cb" (UID: "a95d0313-2000-4c56-a7fa-d4d4a30e23cb"). InnerVolumeSpecName "kube-api-access-d2mnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:24:34.944730 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:34.944649 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:24:34.944730 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:34.944680 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2mnv\" (UniqueName: \"kubernetes.io/projected/a95d0313-2000-4c56-a7fa-d4d4a30e23cb-kube-api-access-d2mnv\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:24:35.654336 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:35.654308 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" Nov 03 10:24:35.654507 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:35.654308 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7" event={"ID":"a95d0313-2000-4c56-a7fa-d4d4a30e23cb","Type":"ContainerDied","Data":"91c093206df57c6d8ed7b5a19207dd55abff736098d01164053157431f42a422"} Nov 03 10:24:35.654507 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:24:35.654453 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91c093206df57c6d8ed7b5a19207dd55abff736098d01164053157431f42a422" Nov 03 10:24:46.901447 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:24:46.901406 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:25:02.054942 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:02.054906 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw"] Nov 03 10:25:02.079540 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:02.079498 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xprdw"] Nov 03 10:25:03.211700 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:03.211664 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b214d719-7cd6-4101-8201-99a0528d2841" path="/var/lib/kubelet/pods/b214d719-7cd6-4101-8201-99a0528d2841/volumes" Nov 03 10:25:16.902003 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:25:16.901912 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:25:19.587866 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.587834 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm"] Nov 03 10:25:19.588313 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.588276 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a95d0313-2000-4c56-a7fa-d4d4a30e23cb" containerName="revoke-keys" Nov 03 10:25:19.588313 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.588289 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95d0313-2000-4c56-a7fa-d4d4a30e23cb" containerName="revoke-keys" Nov 03 10:25:19.588410 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.588368 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a95d0313-2000-4c56-a7fa-d4d4a30e23cb" containerName="revoke-keys" Nov 03 10:25:19.591523 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.591507 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:19.648564 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.648536 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm"] Nov 03 10:25:19.694817 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.694787 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:25:19.694999 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.694900 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:25:19.725550 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.725502 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189eb33c-hj2zm\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:19.725751 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.725618 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189eb33c-hj2zm\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:19.725751 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.725653 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsdxm\" (UniqueName: \"kubernetes.io/projected/355aac11-afb8-4865-a7f1-50db3ec3d7d2-kube-api-access-qsdxm\") pod \"maas-api-revoke-sub-189eb33c-hj2zm\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:19.826767 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.826733 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189eb33c-hj2zm\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:19.826949 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.826799 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189eb33c-hj2zm\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:19.826949 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.826833 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsdxm\" (UniqueName: \"kubernetes.io/projected/355aac11-afb8-4865-a7f1-50db3ec3d7d2-kube-api-access-qsdxm\") pod \"maas-api-revoke-sub-189eb33c-hj2zm\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:19.827616 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.827586 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189eb33c-hj2zm\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:19.829392 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.829358 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189eb33c-hj2zm\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:19.886063 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.885993 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsdxm\" (UniqueName: \"kubernetes.io/projected/355aac11-afb8-4865-a7f1-50db3ec3d7d2-kube-api-access-qsdxm\") pod \"maas-api-revoke-sub-189eb33c-hj2zm\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:19.904534 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:19.904509 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:20.285161 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:20.285130 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm"] Nov 03 10:25:20.286667 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:25:20.286638 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355aac11_afb8_4865_a7f1_50db3ec3d7d2.slice/crio-4dc7d21b3e1938427f088a54262afc66bea51ec13656c620a113443911b5eb52 WatchSource:0}: Error finding container 4dc7d21b3e1938427f088a54262afc66bea51ec13656c620a113443911b5eb52: Status 404 returned error can't find the container with id 4dc7d21b3e1938427f088a54262afc66bea51ec13656c620a113443911b5eb52 Nov 03 10:25:21.069063 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:21.069023 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" event={"ID":"355aac11-afb8-4865-a7f1-50db3ec3d7d2","Type":"ContainerStarted","Data":"5cdb0536c3aac117d2749fc25cd28a441bacf24c80db8b93f447322a308ea3d7"} Nov 03 10:25:21.069534 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:21.069121 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" event={"ID":"355aac11-afb8-4865-a7f1-50db3ec3d7d2","Type":"ContainerStarted","Data":"4dc7d21b3e1938427f088a54262afc66bea51ec13656c620a113443911b5eb52"} Nov 03 10:25:21.187166 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:21.187111 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" podStartSLOduration=2.187097431 podStartE2EDuration="2.187097431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:25:21.185044379 +0000 UTC m=+6354.109986595" watchObservedRunningTime="2026-11-03 10:25:21.187097431 +0000 UTC m=+6354.112039677" Nov 03 10:25:22.076961 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:22.076923 5015 generic.go:356] "Generic (PLEG): container finished" podID="355aac11-afb8-4865-a7f1-50db3ec3d7d2" containerID="5cdb0536c3aac117d2749fc25cd28a441bacf24c80db8b93f447322a308ea3d7" exitCode=0 Nov 03 10:25:22.077420 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:22.077012 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" event={"ID":"355aac11-afb8-4865-a7f1-50db3ec3d7d2","Type":"ContainerDied","Data":"5cdb0536c3aac117d2749fc25cd28a441bacf24c80db8b93f447322a308ea3d7"} Nov 03 10:25:23.234867 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:23.234842 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:23.361067 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:23.360980 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-openshift-ca\") pod \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " Nov 03 10:25:23.361231 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:23.361107 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355aac11-afb8-4865-a7f1-50db3ec3d7d2-kube-api-access-qsdxm\" (UniqueName: \"kubernetes.io/projected/355aac11-afb8-4865-a7f1-50db3ec3d7d2-kube-api-access-qsdxm\") pod \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " Nov 03 10:25:23.361231 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:23.361223 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-xks-ca\") pod \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\" (UID: \"355aac11-afb8-4865-a7f1-50db3ec3d7d2\") " Nov 03 10:25:23.361432 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:23.361403 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-openshift-ca" pod "355aac11-afb8-4865-a7f1-50db3ec3d7d2" (UID: "355aac11-afb8-4865-a7f1-50db3ec3d7d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:25:23.361625 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:23.361604 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:25:23.363513 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:23.363464 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-xks-ca" pod "355aac11-afb8-4865-a7f1-50db3ec3d7d2" (UID: "355aac11-afb8-4865-a7f1-50db3ec3d7d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:25:23.363667 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:23.363644 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355aac11-afb8-4865-a7f1-50db3ec3d7d2-kube-api-access-qsdxm" pod "355aac11-afb8-4865-a7f1-50db3ec3d7d2" (UID: "355aac11-afb8-4865-a7f1-50db3ec3d7d2"). InnerVolumeSpecName "kube-api-access-qsdxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:25:23.462249 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:23.462211 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355aac11-afb8-4865-a7f1-50db3ec3d7d2-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:25:23.462249 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:23.462243 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsdxm\" (UniqueName: \"kubernetes.io/projected/355aac11-afb8-4865-a7f1-50db3ec3d7d2-kube-api-access-qsdxm\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:25:24.090912 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:24.090880 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" Nov 03 10:25:24.090912 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:24.090907 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm" event={"ID":"355aac11-afb8-4865-a7f1-50db3ec3d7d2","Type":"ContainerDied","Data":"4dc7d21b3e1938427f088a54262afc66bea51ec13656c620a113443911b5eb52"} Nov 03 10:25:24.091141 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:24.090960 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc7d21b3e1938427f088a54262afc66bea51ec13656c620a113443911b5eb52" Nov 03 10:25:30.262728 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.262698 5015 scope.go:122] "RemoveContainer" containerID="abade9a71f62b68c97f122e45770898f6bbeaa43611fb385f4ac6b1eb7deaa98" Nov 03 10:25:30.365781 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.365748 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26"] Nov 03 10:25:30.366239 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.366226 5015 cpu_manager.go:414] "containerMap: removing container" podUID="355aac11-afb8-4865-a7f1-50db3ec3d7d2" containerName="revoke-keys" Nov 03 10:25:30.366285 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.366241 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355aac11-afb8-4865-a7f1-50db3ec3d7d2" containerName="revoke-keys" Nov 03 10:25:30.366331 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.366322 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="355aac11-afb8-4865-a7f1-50db3ec3d7d2" containerName="revoke-keys" Nov 03 10:25:30.370786 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.370767 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:30.426704 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.426667 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26"] Nov 03 10:25:30.472907 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.472879 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:25:30.472907 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.472909 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:25:30.535054 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.534953 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0a4d67c-bcn26\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:30.535236 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.535111 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0a4d67c-bcn26\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:30.535289 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.535233 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2gq5\" (UniqueName: \"kubernetes.io/projected/85404c28-3ba7-45f8-a61b-6e726be726e9-kube-api-access-l2gq5\") pod \"maas-api-revoke-sub-d0a4d67c-bcn26\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:30.636724 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.636673 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2gq5\" (UniqueName: \"kubernetes.io/projected/85404c28-3ba7-45f8-a61b-6e726be726e9-kube-api-access-l2gq5\") pod \"maas-api-revoke-sub-d0a4d67c-bcn26\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:30.636724 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.636737 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0a4d67c-bcn26\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:30.636962 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.636808 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0a4d67c-bcn26\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:30.637361 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.637331 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0a4d67c-bcn26\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:30.639322 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.639300 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0a4d67c-bcn26\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:30.697007 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.696967 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2gq5\" (UniqueName: \"kubernetes.io/projected/85404c28-3ba7-45f8-a61b-6e726be726e9-kube-api-access-l2gq5\") pod \"maas-api-revoke-sub-d0a4d67c-bcn26\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:30.984464 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:30.984426 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:31.366405 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:31.366368 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26"] Nov 03 10:25:31.367097 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:25:31.367063 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85404c28_3ba7_45f8_a61b_6e726be726e9.slice/crio-589464092a22f8de4821202bb193904febe3dc3321f2dc4557642d5a676cfd7b WatchSource:0}: Error finding container 589464092a22f8de4821202bb193904febe3dc3321f2dc4557642d5a676cfd7b: Status 404 returned error can't find the container with id 589464092a22f8de4821202bb193904febe3dc3321f2dc4557642d5a676cfd7b Nov 03 10:25:32.142537 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:32.142495 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" event={"ID":"85404c28-3ba7-45f8-a61b-6e726be726e9","Type":"ContainerStarted","Data":"7e0a63bb050e4efa9e8f5b267df4dbc6a51859bcaadf828dfe4cb9463e007161"} Nov 03 10:25:32.142808 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:32.142568 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" event={"ID":"85404c28-3ba7-45f8-a61b-6e726be726e9","Type":"ContainerStarted","Data":"589464092a22f8de4821202bb193904febe3dc3321f2dc4557642d5a676cfd7b"} Nov 03 10:25:32.260467 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:32.260412 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" podStartSLOduration=2.2603969729999998 podStartE2EDuration="2.260396973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:25:32.257255096 +0000 UTC m=+6365.182197313" watchObservedRunningTime="2026-11-03 10:25:32.260396973 +0000 UTC m=+6365.185339190" Nov 03 10:25:33.152900 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:33.152858 5015 generic.go:356] "Generic (PLEG): container finished" podID="85404c28-3ba7-45f8-a61b-6e726be726e9" containerID="7e0a63bb050e4efa9e8f5b267df4dbc6a51859bcaadf828dfe4cb9463e007161" exitCode=0 Nov 03 10:25:33.153607 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:33.152953 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" event={"ID":"85404c28-3ba7-45f8-a61b-6e726be726e9","Type":"ContainerDied","Data":"7e0a63bb050e4efa9e8f5b267df4dbc6a51859bcaadf828dfe4cb9463e007161"} Nov 03 10:25:34.626769 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:34.626728 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:34.724828 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:34.724776 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85404c28-3ba7-45f8-a61b-6e726be726e9-kube-api-access-l2gq5\" (UniqueName: \"kubernetes.io/projected/85404c28-3ba7-45f8-a61b-6e726be726e9-kube-api-access-l2gq5\") pod \"85404c28-3ba7-45f8-a61b-6e726be726e9\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " Nov 03 10:25:34.725047 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:34.724841 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-openshift-ca\") pod \"85404c28-3ba7-45f8-a61b-6e726be726e9\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " Nov 03 10:25:34.725047 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:34.724925 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-xks-ca\") pod \"85404c28-3ba7-45f8-a61b-6e726be726e9\" (UID: \"85404c28-3ba7-45f8-a61b-6e726be726e9\") " Nov 03 10:25:34.725363 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:34.725336 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-openshift-ca" pod "85404c28-3ba7-45f8-a61b-6e726be726e9" (UID: "85404c28-3ba7-45f8-a61b-6e726be726e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:25:34.727338 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:34.727309 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85404c28-3ba7-45f8-a61b-6e726be726e9-kube-api-access-l2gq5" pod "85404c28-3ba7-45f8-a61b-6e726be726e9" (UID: "85404c28-3ba7-45f8-a61b-6e726be726e9"). InnerVolumeSpecName "kube-api-access-l2gq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:25:34.732087 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:34.732064 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-xks-ca" pod "85404c28-3ba7-45f8-a61b-6e726be726e9" (UID: "85404c28-3ba7-45f8-a61b-6e726be726e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:25:34.829568 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:34.829104 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2gq5\" (UniqueName: \"kubernetes.io/projected/85404c28-3ba7-45f8-a61b-6e726be726e9-kube-api-access-l2gq5\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:25:34.829568 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:34.829151 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:25:34.829568 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:34.829168 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85404c28-3ba7-45f8-a61b-6e726be726e9-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:25:35.216245 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:35.216146 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" Nov 03 10:25:35.219537 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:35.219463 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26" event={"ID":"85404c28-3ba7-45f8-a61b-6e726be726e9","Type":"ContainerDied","Data":"589464092a22f8de4821202bb193904febe3dc3321f2dc4557642d5a676cfd7b"} Nov 03 10:25:35.219703 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:25:35.219545 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="589464092a22f8de4821202bb193904febe3dc3321f2dc4557642d5a676cfd7b" Nov 03 10:25:46.901092 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:25:46.901056 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:26:15.514661 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:15.514605 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:26:15.515512 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:15.515444 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:26:15.825699 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:15.825600 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:26:15.825897 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:15.825690 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:26:16.901717 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:26:16.901679 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:26:21.359055 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.359014 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r"] Nov 03 10:26:21.359698 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.359643 5015 cpu_manager.go:414] "containerMap: removing container" podUID="85404c28-3ba7-45f8-a61b-6e726be726e9" containerName="revoke-keys" Nov 03 10:26:21.359698 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.359663 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85404c28-3ba7-45f8-a61b-6e726be726e9" containerName="revoke-keys" Nov 03 10:26:21.359826 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.359816 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="85404c28-3ba7-45f8-a61b-6e726be726e9" containerName="revoke-keys" Nov 03 10:26:21.363648 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.363620 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:21.419494 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.419446 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r"] Nov 03 10:26:21.462601 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.462571 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ckj9\" (UniqueName: \"kubernetes.io/projected/487e8c78-f695-4dcf-aa4e-b8668a69c646-kube-api-access-5ckj9\") pod \"maas-api-revoke-sub-861756ce-tr49r\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:21.462783 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.462629 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-861756ce-tr49r\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:21.462783 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.462684 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-861756ce-tr49r\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:21.465594 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.465573 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:26:21.465690 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.465623 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:26:21.564127 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.564096 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ckj9\" (UniqueName: \"kubernetes.io/projected/487e8c78-f695-4dcf-aa4e-b8668a69c646-kube-api-access-5ckj9\") pod \"maas-api-revoke-sub-861756ce-tr49r\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:21.564298 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.564164 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-861756ce-tr49r\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:21.564298 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.564196 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-861756ce-tr49r\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:21.564828 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.564807 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-861756ce-tr49r\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:21.566748 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.566724 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-861756ce-tr49r\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:21.623846 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.623784 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ckj9\" (UniqueName: \"kubernetes.io/projected/487e8c78-f695-4dcf-aa4e-b8668a69c646-kube-api-access-5ckj9\") pod \"maas-api-revoke-sub-861756ce-tr49r\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:21.680187 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:21.680156 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:22.057591 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:22.057559 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r"] Nov 03 10:26:22.058166 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:26:22.058141 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487e8c78_f695_4dcf_aa4e_b8668a69c646.slice/crio-4152657ea50ba85b9dfb3a197418784b717be8ddf7ce63ad5f189c1e61f6286e WatchSource:0}: Error finding container 4152657ea50ba85b9dfb3a197418784b717be8ddf7ce63ad5f189c1e61f6286e: Status 404 returned error can't find the container with id 4152657ea50ba85b9dfb3a197418784b717be8ddf7ce63ad5f189c1e61f6286e Nov 03 10:26:22.656259 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:22.656172 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" event={"ID":"487e8c78-f695-4dcf-aa4e-b8668a69c646","Type":"ContainerStarted","Data":"d009005cc69a6dced323a556d3dcd5caba6714433189994def540f51cc3795c6"} Nov 03 10:26:22.656259 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:22.656234 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" event={"ID":"487e8c78-f695-4dcf-aa4e-b8668a69c646","Type":"ContainerStarted","Data":"4152657ea50ba85b9dfb3a197418784b717be8ddf7ce63ad5f189c1e61f6286e"} Nov 03 10:26:22.772379 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:22.772323 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" podStartSLOduration=1.772308373 podStartE2EDuration="1.772308373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:26:22.77078063 +0000 UTC m=+6415.695722872" watchObservedRunningTime="2026-11-03 10:26:22.772308373 +0000 UTC m=+6415.697250590" Nov 03 10:26:23.663782 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:23.663742 5015 generic.go:356] "Generic (PLEG): container finished" podID="487e8c78-f695-4dcf-aa4e-b8668a69c646" containerID="d009005cc69a6dced323a556d3dcd5caba6714433189994def540f51cc3795c6" exitCode=0 Nov 03 10:26:23.664251 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:23.663828 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" event={"ID":"487e8c78-f695-4dcf-aa4e-b8668a69c646","Type":"ContainerDied","Data":"d009005cc69a6dced323a556d3dcd5caba6714433189994def540f51cc3795c6"} Nov 03 10:26:24.811997 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:24.811971 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:24.897046 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:24.897009 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487e8c78-f695-4dcf-aa4e-b8668a69c646-kube-api-access-5ckj9\" (UniqueName: \"kubernetes.io/projected/487e8c78-f695-4dcf-aa4e-b8668a69c646-kube-api-access-5ckj9\") pod \"487e8c78-f695-4dcf-aa4e-b8668a69c646\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " Nov 03 10:26:24.897225 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:24.897114 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-xks-ca\") pod \"487e8c78-f695-4dcf-aa4e-b8668a69c646\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " Nov 03 10:26:24.897225 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:24.897151 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-openshift-ca\") pod \"487e8c78-f695-4dcf-aa4e-b8668a69c646\" (UID: \"487e8c78-f695-4dcf-aa4e-b8668a69c646\") " Nov 03 10:26:24.897581 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:24.897555 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-openshift-ca" pod "487e8c78-f695-4dcf-aa4e-b8668a69c646" (UID: "487e8c78-f695-4dcf-aa4e-b8668a69c646"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:26:24.899525 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:24.899503 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487e8c78-f695-4dcf-aa4e-b8668a69c646-kube-api-access-5ckj9" pod "487e8c78-f695-4dcf-aa4e-b8668a69c646" (UID: "487e8c78-f695-4dcf-aa4e-b8668a69c646"). InnerVolumeSpecName "kube-api-access-5ckj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:26:24.899525 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:24.899511 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-xks-ca" pod "487e8c78-f695-4dcf-aa4e-b8668a69c646" (UID: "487e8c78-f695-4dcf-aa4e-b8668a69c646"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:26:24.998112 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:24.998081 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:24.998112 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:24.998110 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487e8c78-f695-4dcf-aa4e-b8668a69c646-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:24.998321 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:24.998121 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ckj9\" (UniqueName: \"kubernetes.io/projected/487e8c78-f695-4dcf-aa4e-b8668a69c646-kube-api-access-5ckj9\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:25.677565 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:25.677528 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" event={"ID":"487e8c78-f695-4dcf-aa4e-b8668a69c646","Type":"ContainerDied","Data":"4152657ea50ba85b9dfb3a197418784b717be8ddf7ce63ad5f189c1e61f6286e"} Nov 03 10:26:25.677565 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:25.677549 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r" Nov 03 10:26:25.677831 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:25.677582 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4152657ea50ba85b9dfb3a197418784b717be8ddf7ce63ad5f189c1e61f6286e" Nov 03 10:26:30.290600 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:30.290568 5015 scope.go:122] "RemoveContainer" containerID="bb6aae74513a9b8a46d1f1cb1d177c17fed2d7c194e15dfb5bce573a65af1531" Nov 03 10:26:32.594607 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.594571 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml"] Nov 03 10:26:32.595057 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.595043 5015 cpu_manager.go:414] "containerMap: removing container" podUID="487e8c78-f695-4dcf-aa4e-b8668a69c646" containerName="revoke-keys" Nov 03 10:26:32.595105 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.595059 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487e8c78-f695-4dcf-aa4e-b8668a69c646" containerName="revoke-keys" Nov 03 10:26:32.595146 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.595141 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="487e8c78-f695-4dcf-aa4e-b8668a69c646" containerName="revoke-keys" Nov 03 10:26:32.599666 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.599645 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:32.658969 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.658937 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml"] Nov 03 10:26:32.677170 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.677133 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0fd88da-c4bml\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:32.677360 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.677175 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shb2f\" (UniqueName: \"kubernetes.io/projected/72de1663-8c9b-4c9b-b186-8e2872f1622b-kube-api-access-shb2f\") pod \"maas-api-revoke-sub-d0fd88da-c4bml\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:32.677360 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.677256 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0fd88da-c4bml\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:32.703753 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.703724 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:26:32.703856 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.703782 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:26:32.778228 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.778189 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0fd88da-c4bml\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:32.778422 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.778237 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shb2f\" (UniqueName: \"kubernetes.io/projected/72de1663-8c9b-4c9b-b186-8e2872f1622b-kube-api-access-shb2f\") pod \"maas-api-revoke-sub-d0fd88da-c4bml\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:32.778422 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.778282 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0fd88da-c4bml\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:32.778930 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.778904 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0fd88da-c4bml\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:32.780854 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.780833 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0fd88da-c4bml\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:32.837552 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.837523 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shb2f\" (UniqueName: \"kubernetes.io/projected/72de1663-8c9b-4c9b-b186-8e2872f1622b-kube-api-access-shb2f\") pod \"maas-api-revoke-sub-d0fd88da-c4bml\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:32.916612 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:32.916537 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:33.296287 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:33.296257 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml"] Nov 03 10:26:33.297356 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:26:33.297326 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72de1663_8c9b_4c9b_b186_8e2872f1622b.slice/crio-ed2ff6cb7bd4da674f579d64354d3fd81eefc6114408a465c099f997b1c92930 WatchSource:0}: Error finding container ed2ff6cb7bd4da674f579d64354d3fd81eefc6114408a465c099f997b1c92930: Status 404 returned error can't find the container with id ed2ff6cb7bd4da674f579d64354d3fd81eefc6114408a465c099f997b1c92930 Nov 03 10:26:33.731300 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:33.731259 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" event={"ID":"72de1663-8c9b-4c9b-b186-8e2872f1622b","Type":"ContainerStarted","Data":"b36faa9d1cd36c1381f3a44cf39477c124d191e4bb9e564bf4d3c3270c9dc91f"} Nov 03 10:26:33.731739 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:33.731321 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" event={"ID":"72de1663-8c9b-4c9b-b186-8e2872f1622b","Type":"ContainerStarted","Data":"ed2ff6cb7bd4da674f579d64354d3fd81eefc6114408a465c099f997b1c92930"} Nov 03 10:26:33.850396 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:33.850291 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" podStartSLOduration=1.850275645 podStartE2EDuration="1.850275645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:26:33.846765067 +0000 UTC m=+6426.771707286" watchObservedRunningTime="2026-11-03 10:26:33.850275645 +0000 UTC m=+6426.775217859" Nov 03 10:26:34.738812 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:34.738778 5015 generic.go:356] "Generic (PLEG): container finished" podID="72de1663-8c9b-4c9b-b186-8e2872f1622b" containerID="b36faa9d1cd36c1381f3a44cf39477c124d191e4bb9e564bf4d3c3270c9dc91f" exitCode=0 Nov 03 10:26:34.739281 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:34.738843 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" event={"ID":"72de1663-8c9b-4c9b-b186-8e2872f1622b","Type":"ContainerDied","Data":"b36faa9d1cd36c1381f3a44cf39477c124d191e4bb9e564bf4d3c3270c9dc91f"} Nov 03 10:26:35.909220 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:35.909186 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:36.013613 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.013525 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-xks-ca\") pod \"72de1663-8c9b-4c9b-b186-8e2872f1622b\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " Nov 03 10:26:36.013782 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.013623 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72de1663-8c9b-4c9b-b186-8e2872f1622b-kube-api-access-shb2f\" (UniqueName: \"kubernetes.io/projected/72de1663-8c9b-4c9b-b186-8e2872f1622b-kube-api-access-shb2f\") pod \"72de1663-8c9b-4c9b-b186-8e2872f1622b\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " Nov 03 10:26:36.013782 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.013680 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-openshift-ca\") pod \"72de1663-8c9b-4c9b-b186-8e2872f1622b\" (UID: \"72de1663-8c9b-4c9b-b186-8e2872f1622b\") " Nov 03 10:26:36.014096 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.014052 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-openshift-ca" pod "72de1663-8c9b-4c9b-b186-8e2872f1622b" (UID: "72de1663-8c9b-4c9b-b186-8e2872f1622b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:26:36.015925 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.015897 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-xks-ca" pod "72de1663-8c9b-4c9b-b186-8e2872f1622b" (UID: "72de1663-8c9b-4c9b-b186-8e2872f1622b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:26:36.016029 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.015951 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72de1663-8c9b-4c9b-b186-8e2872f1622b-kube-api-access-shb2f" pod "72de1663-8c9b-4c9b-b186-8e2872f1622b" (UID: "72de1663-8c9b-4c9b-b186-8e2872f1622b"). InnerVolumeSpecName "kube-api-access-shb2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:26:36.115538 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.115495 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:36.115538 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.115532 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shb2f\" (UniqueName: \"kubernetes.io/projected/72de1663-8c9b-4c9b-b186-8e2872f1622b-kube-api-access-shb2f\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:36.115761 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.115548 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72de1663-8c9b-4c9b-b186-8e2872f1622b-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:36.755110 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.755071 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" event={"ID":"72de1663-8c9b-4c9b-b186-8e2872f1622b","Type":"ContainerDied","Data":"ed2ff6cb7bd4da674f579d64354d3fd81eefc6114408a465c099f997b1c92930"} Nov 03 10:26:36.755313 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.755123 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed2ff6cb7bd4da674f579d64354d3fd81eefc6114408a465c099f997b1c92930" Nov 03 10:26:36.755313 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:36.755172 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml" Nov 03 10:26:43.734123 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.734074 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh"] Nov 03 10:26:43.734806 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.734786 5015 cpu_manager.go:414] "containerMap: removing container" podUID="72de1663-8c9b-4c9b-b186-8e2872f1622b" containerName="revoke-keys" Nov 03 10:26:43.734862 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.734810 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72de1663-8c9b-4c9b-b186-8e2872f1622b" containerName="revoke-keys" Nov 03 10:26:43.734954 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.734940 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="72de1663-8c9b-4c9b-b186-8e2872f1622b" containerName="revoke-keys" Nov 03 10:26:43.738721 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.738703 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:43.793731 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.793699 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh"] Nov 03 10:26:43.813048 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.813005 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xhmk\" (UniqueName: \"kubernetes.io/projected/cdc612f6-eb87-424c-ad70-07b5581e27a4-kube-api-access-5xhmk\") pod \"maas-api-revoke-sub-1c9aafc4-xp7hh\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:43.813261 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.813161 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9aafc4-xp7hh\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:43.813261 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.813207 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9aafc4-xp7hh\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:43.839645 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.839608 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:26:43.839796 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.839657 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:26:43.914568 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.914508 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9aafc4-xp7hh\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:43.914781 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.914598 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9aafc4-xp7hh\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:43.914781 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.914726 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhmk\" (UniqueName: \"kubernetes.io/projected/cdc612f6-eb87-424c-ad70-07b5581e27a4-kube-api-access-5xhmk\") pod \"maas-api-revoke-sub-1c9aafc4-xp7hh\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:43.915314 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.915290 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9aafc4-xp7hh\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:43.917150 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.917129 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9aafc4-xp7hh\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:43.976046 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:43.975882 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhmk\" (UniqueName: \"kubernetes.io/projected/cdc612f6-eb87-424c-ad70-07b5581e27a4-kube-api-access-5xhmk\") pod \"maas-api-revoke-sub-1c9aafc4-xp7hh\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:44.055682 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:44.055594 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:44.436640 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:44.436605 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh"] Nov 03 10:26:44.437707 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:26:44.437678 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc612f6_eb87_424c_ad70_07b5581e27a4.slice/crio-4531f838b81c025e606ada6ef91ea7529fb1a60b8af8d37b3bc1a9d1696c176b WatchSource:0}: Error finding container 4531f838b81c025e606ada6ef91ea7529fb1a60b8af8d37b3bc1a9d1696c176b: Status 404 returned error can't find the container with id 4531f838b81c025e606ada6ef91ea7529fb1a60b8af8d37b3bc1a9d1696c176b Nov 03 10:26:44.846494 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:44.846435 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" event={"ID":"cdc612f6-eb87-424c-ad70-07b5581e27a4","Type":"ContainerStarted","Data":"cee01cf0730c04716a9a210b837eb33878e6f2b75b723404881fbe43e885f334"} Nov 03 10:26:44.846967 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:44.846519 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" event={"ID":"cdc612f6-eb87-424c-ad70-07b5581e27a4","Type":"ContainerStarted","Data":"4531f838b81c025e606ada6ef91ea7529fb1a60b8af8d37b3bc1a9d1696c176b"} Nov 03 10:26:44.967454 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:44.967394 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" podStartSLOduration=1.967379678 podStartE2EDuration="1.967379678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:26:44.964170171 +0000 UTC m=+6437.889112413" watchObservedRunningTime="2026-11-03 10:26:44.967379678 +0000 UTC m=+6437.892321889" Nov 03 10:26:45.853654 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:45.853609 5015 generic.go:356] "Generic (PLEG): container finished" podID="cdc612f6-eb87-424c-ad70-07b5581e27a4" containerID="cee01cf0730c04716a9a210b837eb33878e6f2b75b723404881fbe43e885f334" exitCode=0 Nov 03 10:26:45.854114 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:45.853697 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" event={"ID":"cdc612f6-eb87-424c-ad70-07b5581e27a4","Type":"ContainerDied","Data":"cee01cf0730c04716a9a210b837eb33878e6f2b75b723404881fbe43e885f334"} Nov 03 10:26:46.905603 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:26:46.905564 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:26:47.008086 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.008063 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:47.147925 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.147830 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-openshift-ca\") pod \"cdc612f6-eb87-424c-ad70-07b5581e27a4\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " Nov 03 10:26:47.148094 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.147988 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-xks-ca\") pod \"cdc612f6-eb87-424c-ad70-07b5581e27a4\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " Nov 03 10:26:47.148094 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.148025 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc612f6-eb87-424c-ad70-07b5581e27a4-kube-api-access-5xhmk\" (UniqueName: \"kubernetes.io/projected/cdc612f6-eb87-424c-ad70-07b5581e27a4-kube-api-access-5xhmk\") pod \"cdc612f6-eb87-424c-ad70-07b5581e27a4\" (UID: \"cdc612f6-eb87-424c-ad70-07b5581e27a4\") " Nov 03 10:26:47.148269 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.148244 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-openshift-ca" pod "cdc612f6-eb87-424c-ad70-07b5581e27a4" (UID: "cdc612f6-eb87-424c-ad70-07b5581e27a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:26:47.148368 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.148354 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:47.150370 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.150337 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-xks-ca" pod "cdc612f6-eb87-424c-ad70-07b5581e27a4" (UID: "cdc612f6-eb87-424c-ad70-07b5581e27a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:26:47.150597 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.150526 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc612f6-eb87-424c-ad70-07b5581e27a4-kube-api-access-5xhmk" pod "cdc612f6-eb87-424c-ad70-07b5581e27a4" (UID: "cdc612f6-eb87-424c-ad70-07b5581e27a4"). InnerVolumeSpecName "kube-api-access-5xhmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:26:47.249575 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.249541 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc612f6-eb87-424c-ad70-07b5581e27a4-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:47.249575 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.249570 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xhmk\" (UniqueName: \"kubernetes.io/projected/cdc612f6-eb87-424c-ad70-07b5581e27a4-kube-api-access-5xhmk\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:47.867277 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.867243 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" event={"ID":"cdc612f6-eb87-424c-ad70-07b5581e27a4","Type":"ContainerDied","Data":"4531f838b81c025e606ada6ef91ea7529fb1a60b8af8d37b3bc1a9d1696c176b"} Nov 03 10:26:47.867277 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.867279 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh" Nov 03 10:26:47.867526 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:47.867291 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4531f838b81c025e606ada6ef91ea7529fb1a60b8af8d37b3bc1a9d1696c176b" Nov 03 10:26:51.052941 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:51.052907 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g"] Nov 03 10:26:51.101739 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:51.101700 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-x2g7g"] Nov 03 10:26:51.203119 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:51.203075 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f24bda-91fa-44fa-b39d-62087df957f9" path="/var/lib/kubelet/pods/50f24bda-91fa-44fa-b39d-62087df957f9/volumes" Nov 03 10:26:54.665441 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.665401 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt"] Nov 03 10:26:54.666064 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.666043 5015 cpu_manager.go:414] "containerMap: removing container" podUID="cdc612f6-eb87-424c-ad70-07b5581e27a4" containerName="revoke-keys" Nov 03 10:26:54.666162 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.666068 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc612f6-eb87-424c-ad70-07b5581e27a4" containerName="revoke-keys" Nov 03 10:26:54.666255 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.666241 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc612f6-eb87-424c-ad70-07b5581e27a4" containerName="revoke-keys" Nov 03 10:26:54.669747 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.669723 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:54.726308 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.726268 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddw4p\" (UniqueName: \"kubernetes.io/projected/87493051-b17d-4aaa-8674-dbf6575b059d-kube-api-access-ddw4p\") pod \"maas-api-revoke-sub-3028a06a-299zt\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:54.726584 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.726320 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3028a06a-299zt\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:54.726584 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.726490 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3028a06a-299zt\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:54.727014 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.726991 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt"] Nov 03 10:26:54.773201 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.773161 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:26:54.773381 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.773340 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:26:54.827070 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.827030 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3028a06a-299zt\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:54.827257 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.827085 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddw4p\" (UniqueName: \"kubernetes.io/projected/87493051-b17d-4aaa-8674-dbf6575b059d-kube-api-access-ddw4p\") pod \"maas-api-revoke-sub-3028a06a-299zt\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:54.827257 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.827112 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3028a06a-299zt\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:54.827830 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.827807 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3028a06a-299zt\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:54.829811 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.829781 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3028a06a-299zt\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:54.886439 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.886404 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddw4p\" (UniqueName: \"kubernetes.io/projected/87493051-b17d-4aaa-8674-dbf6575b059d-kube-api-access-ddw4p\") pod \"maas-api-revoke-sub-3028a06a-299zt\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:54.982875 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:54.982827 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:55.360489 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:26:55.360439 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87493051_b17d_4aaa_8674_dbf6575b059d.slice/crio-50038b5f8b9ab62822286de1614b999b4e9045ea758d4bc2e6f7c8f341b860f8 WatchSource:0}: Error finding container 50038b5f8b9ab62822286de1614b999b4e9045ea758d4bc2e6f7c8f341b860f8: Status 404 returned error can't find the container with id 50038b5f8b9ab62822286de1614b999b4e9045ea758d4bc2e6f7c8f341b860f8 Nov 03 10:26:55.361464 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:55.361433 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt"] Nov 03 10:26:55.915831 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:55.915732 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" event={"ID":"87493051-b17d-4aaa-8674-dbf6575b059d","Type":"ContainerStarted","Data":"c10420dbb4d1dab4835959f5503a2aa042087cd6cf75d604362b0d97811db7cd"} Nov 03 10:26:55.915831 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:55.915827 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" event={"ID":"87493051-b17d-4aaa-8674-dbf6575b059d","Type":"ContainerStarted","Data":"50038b5f8b9ab62822286de1614b999b4e9045ea758d4bc2e6f7c8f341b860f8"} Nov 03 10:26:56.033345 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:56.033288 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" podStartSLOduration=2.033273108 podStartE2EDuration="2.033273108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:26:56.030386759 +0000 UTC m=+6448.955328995" watchObservedRunningTime="2026-11-03 10:26:56.033273108 +0000 UTC m=+6448.958215324" Nov 03 10:26:56.923445 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:56.923407 5015 generic.go:356] "Generic (PLEG): container finished" podID="87493051-b17d-4aaa-8674-dbf6575b059d" containerID="c10420dbb4d1dab4835959f5503a2aa042087cd6cf75d604362b0d97811db7cd" exitCode=0 Nov 03 10:26:56.923878 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:56.923508 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" event={"ID":"87493051-b17d-4aaa-8674-dbf6575b059d","Type":"ContainerDied","Data":"c10420dbb4d1dab4835959f5503a2aa042087cd6cf75d604362b0d97811db7cd"} Nov 03 10:26:58.115227 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.115200 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:58.265703 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.265672 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-xks-ca\") pod \"87493051-b17d-4aaa-8674-dbf6575b059d\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " Nov 03 10:26:58.265934 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.265719 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-openshift-ca\") pod \"87493051-b17d-4aaa-8674-dbf6575b059d\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " Nov 03 10:26:58.265934 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.265748 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87493051-b17d-4aaa-8674-dbf6575b059d-kube-api-access-ddw4p\" (UniqueName: \"kubernetes.io/projected/87493051-b17d-4aaa-8674-dbf6575b059d-kube-api-access-ddw4p\") pod \"87493051-b17d-4aaa-8674-dbf6575b059d\" (UID: \"87493051-b17d-4aaa-8674-dbf6575b059d\") " Nov 03 10:26:58.266257 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.266228 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-openshift-ca" pod "87493051-b17d-4aaa-8674-dbf6575b059d" (UID: "87493051-b17d-4aaa-8674-dbf6575b059d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:26:58.268326 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.268283 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87493051-b17d-4aaa-8674-dbf6575b059d-kube-api-access-ddw4p" pod "87493051-b17d-4aaa-8674-dbf6575b059d" (UID: "87493051-b17d-4aaa-8674-dbf6575b059d"). InnerVolumeSpecName "kube-api-access-ddw4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:26:58.268712 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.268684 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-xks-ca" pod "87493051-b17d-4aaa-8674-dbf6575b059d" (UID: "87493051-b17d-4aaa-8674-dbf6575b059d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:26:58.366744 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.366705 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:58.366744 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.366741 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87493051-b17d-4aaa-8674-dbf6575b059d-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:58.366956 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.366756 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddw4p\" (UniqueName: \"kubernetes.io/projected/87493051-b17d-4aaa-8674-dbf6575b059d-kube-api-access-ddw4p\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:26:58.940235 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.940202 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" Nov 03 10:26:58.940235 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.940225 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt" event={"ID":"87493051-b17d-4aaa-8674-dbf6575b059d","Type":"ContainerDied","Data":"50038b5f8b9ab62822286de1614b999b4e9045ea758d4bc2e6f7c8f341b860f8"} Nov 03 10:26:58.940557 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:26:58.940272 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50038b5f8b9ab62822286de1614b999b4e9045ea758d4bc2e6f7c8f341b860f8" Nov 03 10:27:01.769166 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:01.769114 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:27:01.769913 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:01.769200 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lkg9f" podUID="fdb2f094-67e8-438b-80b8-c1ef4521b355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:27:05.359102 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.359062 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq"] Nov 03 10:27:05.359722 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.359702 5015 cpu_manager.go:414] "containerMap: removing container" podUID="87493051-b17d-4aaa-8674-dbf6575b059d" containerName="revoke-keys" Nov 03 10:27:05.359819 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.359724 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87493051-b17d-4aaa-8674-dbf6575b059d" containerName="revoke-keys" Nov 03 10:27:05.359879 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.359844 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="87493051-b17d-4aaa-8674-dbf6575b059d" containerName="revoke-keys" Nov 03 10:27:05.363330 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.363312 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:05.419713 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.419679 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq"] Nov 03 10:27:05.452056 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.452017 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxsfq\" (UniqueName: \"kubernetes.io/projected/249c8c10-8e03-41c6-997a-c413fff553e0-kube-api-access-bxsfq\") pod \"maas-api-revoke-sub-11318af9-rnqdq\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:05.452233 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.452076 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11318af9-rnqdq\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:05.452233 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.452165 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11318af9-rnqdq\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:05.465530 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.465505 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:27:05.465530 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.465506 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:27:05.553244 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.553216 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxsfq\" (UniqueName: \"kubernetes.io/projected/249c8c10-8e03-41c6-997a-c413fff553e0-kube-api-access-bxsfq\") pod \"maas-api-revoke-sub-11318af9-rnqdq\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:05.553405 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.553257 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11318af9-rnqdq\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:05.553405 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.553282 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11318af9-rnqdq\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:05.553927 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.553906 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11318af9-rnqdq\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:05.555910 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.555889 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11318af9-rnqdq\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:05.612142 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.612074 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxsfq\" (UniqueName: \"kubernetes.io/projected/249c8c10-8e03-41c6-997a-c413fff553e0-kube-api-access-bxsfq\") pod \"maas-api-revoke-sub-11318af9-rnqdq\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:05.677153 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:05.677123 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:06.052486 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:06.052445 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq"] Nov 03 10:27:06.053950 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:27:06.053913 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249c8c10_8e03_41c6_997a_c413fff553e0.slice/crio-29b8307f41046dfa9617bb3cb78912aeed38e616a410e3c393f543bb5dc13afb WatchSource:0}: Error finding container 29b8307f41046dfa9617bb3cb78912aeed38e616a410e3c393f543bb5dc13afb: Status 404 returned error can't find the container with id 29b8307f41046dfa9617bb3cb78912aeed38e616a410e3c393f543bb5dc13afb Nov 03 10:27:07.019096 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:07.019057 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" event={"ID":"249c8c10-8e03-41c6-997a-c413fff553e0","Type":"ContainerStarted","Data":"dad1a27d960bcddc8a0a572c61bb92914da02d25939b410211c2d730962e120f"} Nov 03 10:27:07.019653 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:07.019120 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" event={"ID":"249c8c10-8e03-41c6-997a-c413fff553e0","Type":"ContainerStarted","Data":"29b8307f41046dfa9617bb3cb78912aeed38e616a410e3c393f543bb5dc13afb"} Nov 03 10:27:07.136426 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:07.136375 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" podStartSLOduration=2.136360697 podStartE2EDuration="2.136360697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:27:07.134036263 +0000 UTC m=+6460.058978490" watchObservedRunningTime="2026-11-03 10:27:07.136360697 +0000 UTC m=+6460.061302914" Nov 03 10:27:08.027050 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:08.027013 5015 generic.go:356] "Generic (PLEG): container finished" podID="249c8c10-8e03-41c6-997a-c413fff553e0" containerID="dad1a27d960bcddc8a0a572c61bb92914da02d25939b410211c2d730962e120f" exitCode=0 Nov 03 10:27:08.027499 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:08.027103 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" event={"ID":"249c8c10-8e03-41c6-997a-c413fff553e0","Type":"ContainerDied","Data":"dad1a27d960bcddc8a0a572c61bb92914da02d25939b410211c2d730962e120f"} Nov 03 10:27:09.178443 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:09.178414 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:09.190632 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:09.190611 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/249c8c10-8e03-41c6-997a-c413fff553e0-kube-api-access-bxsfq\" (UniqueName: \"kubernetes.io/projected/249c8c10-8e03-41c6-997a-c413fff553e0-kube-api-access-bxsfq\") pod \"249c8c10-8e03-41c6-997a-c413fff553e0\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " Nov 03 10:27:09.190739 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:09.190723 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-openshift-ca\") pod \"249c8c10-8e03-41c6-997a-c413fff553e0\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " Nov 03 10:27:09.190781 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:09.190751 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-xks-ca\") pod \"249c8c10-8e03-41c6-997a-c413fff553e0\" (UID: \"249c8c10-8e03-41c6-997a-c413fff553e0\") " Nov 03 10:27:09.191108 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:09.191085 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-openshift-ca" pod "249c8c10-8e03-41c6-997a-c413fff553e0" (UID: "249c8c10-8e03-41c6-997a-c413fff553e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:27:09.192803 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:09.192776 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249c8c10-8e03-41c6-997a-c413fff553e0-kube-api-access-bxsfq" pod "249c8c10-8e03-41c6-997a-c413fff553e0" (UID: "249c8c10-8e03-41c6-997a-c413fff553e0"). InnerVolumeSpecName "kube-api-access-bxsfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:27:09.193008 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:09.192984 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-xks-ca" pod "249c8c10-8e03-41c6-997a-c413fff553e0" (UID: "249c8c10-8e03-41c6-997a-c413fff553e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:27:09.292662 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:09.292582 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:09.292662 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:09.292609 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/249c8c10-8e03-41c6-997a-c413fff553e0-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:09.292662 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:09.292620 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxsfq\" (UniqueName: \"kubernetes.io/projected/249c8c10-8e03-41c6-997a-c413fff553e0-kube-api-access-bxsfq\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:10.042247 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:10.042207 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" event={"ID":"249c8c10-8e03-41c6-997a-c413fff553e0","Type":"ContainerDied","Data":"29b8307f41046dfa9617bb3cb78912aeed38e616a410e3c393f543bb5dc13afb"} Nov 03 10:27:10.042247 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:10.042226 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq" Nov 03 10:27:10.042462 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:10.042261 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29b8307f41046dfa9617bb3cb78912aeed38e616a410e3c393f543bb5dc13afb" Nov 03 10:27:12.060849 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:12.060812 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg"] Nov 03 10:27:12.102965 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:12.102931 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4t6bg"] Nov 03 10:27:13.203092 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:13.203060 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81" path="/var/lib/kubelet/pods/b3a2fee7-c9e8-4fc5-a1c5-ac637b100c81/volumes" Nov 03 10:27:16.462069 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.462025 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks"] Nov 03 10:27:16.462702 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.462682 5015 cpu_manager.go:414] "containerMap: removing container" podUID="249c8c10-8e03-41c6-997a-c413fff553e0" containerName="revoke-keys" Nov 03 10:27:16.462804 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.462704 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249c8c10-8e03-41c6-997a-c413fff553e0" containerName="revoke-keys" Nov 03 10:27:16.462870 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.462813 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="249c8c10-8e03-41c6-997a-c413fff553e0" containerName="revoke-keys" Nov 03 10:27:16.466366 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.466345 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:16.522818 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.522794 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks"] Nov 03 10:27:16.567168 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.567140 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5557744-rr2ks\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:16.567311 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.567197 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5557744-rr2ks\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:16.567311 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.567282 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2lxb\" (UniqueName: \"kubernetes.io/projected/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-kube-api-access-d2lxb\") pod \"maas-api-revoke-sub-c5557744-rr2ks\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:16.568721 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.568700 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:27:16.568800 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.568750 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:27:16.668216 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.668183 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5557744-rr2ks\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:16.668401 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.668224 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2lxb\" (UniqueName: \"kubernetes.io/projected/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-kube-api-access-d2lxb\") pod \"maas-api-revoke-sub-c5557744-rr2ks\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:16.668401 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.668334 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5557744-rr2ks\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:16.669013 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.668988 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5557744-rr2ks\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:16.670914 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.670894 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5557744-rr2ks\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:16.727597 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.727561 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2lxb\" (UniqueName: \"kubernetes.io/projected/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-kube-api-access-d2lxb\") pod \"maas-api-revoke-sub-c5557744-rr2ks\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:16.780076 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:16.780045 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:16.902060 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:27:16.902022 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:27:17.159494 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:17.159449 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks"] Nov 03 10:27:17.159818 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:27:17.159799 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1c33e6_6e6c_4676_b84e_4a2226d1bb4a.slice/crio-a7b5bb02e674432cbb7551dd4bad81631b8289b55ec5507a62d78af4fbd4593c WatchSource:0}: Error finding container a7b5bb02e674432cbb7551dd4bad81631b8289b55ec5507a62d78af4fbd4593c: Status 404 returned error can't find the container with id a7b5bb02e674432cbb7551dd4bad81631b8289b55ec5507a62d78af4fbd4593c Nov 03 10:27:18.101651 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:18.101616 5015 generic.go:356] "Generic (PLEG): container finished" podID="8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a" containerID="280d073c58446dd517e3e297c8b682024b00628a75256820404b2441ed5a88d3" exitCode=0 Nov 03 10:27:18.102066 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:18.101704 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" event={"ID":"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a","Type":"ContainerDied","Data":"280d073c58446dd517e3e297c8b682024b00628a75256820404b2441ed5a88d3"} Nov 03 10:27:18.102066 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:18.101776 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" event={"ID":"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a","Type":"ContainerStarted","Data":"a7b5bb02e674432cbb7551dd4bad81631b8289b55ec5507a62d78af4fbd4593c"} Nov 03 10:27:19.251963 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:19.251939 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:19.396388 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:19.396312 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-xks-ca\") pod \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " Nov 03 10:27:19.396388 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:19.396359 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-openshift-ca\") pod \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " Nov 03 10:27:19.396601 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:19.396466 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-kube-api-access-d2lxb\" (UniqueName: \"kubernetes.io/projected/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-kube-api-access-d2lxb\") pod \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\" (UID: \"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a\") " Nov 03 10:27:19.396823 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:19.396788 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-openshift-ca" pod "8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a" (UID: "8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:27:19.398825 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:19.398796 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-xks-ca" pod "8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a" (UID: "8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:27:19.398825 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:19.398812 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-kube-api-access-d2lxb" pod "8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a" (UID: "8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a"). InnerVolumeSpecName "kube-api-access-d2lxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:27:19.498347 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:19.498301 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2lxb\" (UniqueName: \"kubernetes.io/projected/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-kube-api-access-d2lxb\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:19.498347 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:19.498346 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:19.498583 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:19.498362 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:20.117788 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:20.117762 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" Nov 03 10:27:20.117788 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:20.117781 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks" event={"ID":"8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a","Type":"ContainerDied","Data":"a7b5bb02e674432cbb7551dd4bad81631b8289b55ec5507a62d78af4fbd4593c"} Nov 03 10:27:20.118010 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:20.117825 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7b5bb02e674432cbb7551dd4bad81631b8289b55ec5507a62d78af4fbd4593c" Nov 03 10:27:27.186881 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.186847 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc"] Nov 03 10:27:27.187374 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.187287 5015 cpu_manager.go:414] "containerMap: removing container" podUID="8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a" containerName="revoke-keys" Nov 03 10:27:27.187374 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.187298 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a" containerName="revoke-keys" Nov 03 10:27:27.187374 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.187371 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a" containerName="revoke-keys" Nov 03 10:27:27.190643 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.190622 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:27.247101 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.247073 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc"] Nov 03 10:27:27.275243 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.275210 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4q59\" (UniqueName: \"kubernetes.io/projected/10eabecf-4c83-4930-b92a-e478440f283d-kube-api-access-s4q59\") pod \"maas-api-revoke-sub-06b0b816-sfmjc\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:27.275424 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.275257 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06b0b816-sfmjc\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:27.275424 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.275278 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06b0b816-sfmjc\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:27.293649 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.293625 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:27:27.293831 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.293815 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:27:27.376791 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.376755 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4q59\" (UniqueName: \"kubernetes.io/projected/10eabecf-4c83-4930-b92a-e478440f283d-kube-api-access-s4q59\") pod \"maas-api-revoke-sub-06b0b816-sfmjc\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:27.376976 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.376818 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06b0b816-sfmjc\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:27.376976 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.376851 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06b0b816-sfmjc\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:27.377542 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.377522 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06b0b816-sfmjc\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:27.379393 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.379374 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06b0b816-sfmjc\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:27.436148 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.436122 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4q59\" (UniqueName: \"kubernetes.io/projected/10eabecf-4c83-4930-b92a-e478440f283d-kube-api-access-s4q59\") pod \"maas-api-revoke-sub-06b0b816-sfmjc\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:27.506849 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.506819 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:27.886169 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:27.886133 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc"] Nov 03 10:27:27.886586 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:27:27.886562 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10eabecf_4c83_4930_b92a_e478440f283d.slice/crio-96dbd705dde798214ece74b69d868338651e129a7fee68729e9e22a33a53f306 WatchSource:0}: Error finding container 96dbd705dde798214ece74b69d868338651e129a7fee68729e9e22a33a53f306: Status 404 returned error can't find the container with id 96dbd705dde798214ece74b69d868338651e129a7fee68729e9e22a33a53f306 Nov 03 10:27:28.171415 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:28.171381 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" event={"ID":"10eabecf-4c83-4930-b92a-e478440f283d","Type":"ContainerStarted","Data":"436ecd695b07986d48ff6c4e156a24cc1f62201a812d78f8f9dfa2e79f13a222"} Nov 03 10:27:28.171586 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:28.171436 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" event={"ID":"10eabecf-4c83-4930-b92a-e478440f283d","Type":"ContainerStarted","Data":"96dbd705dde798214ece74b69d868338651e129a7fee68729e9e22a33a53f306"} Nov 03 10:27:28.289741 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:28.289690 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" podStartSLOduration=1.289677245 podStartE2EDuration="1.289677245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:27:28.285523012 +0000 UTC m=+6481.210465223" watchObservedRunningTime="2026-11-03 10:27:28.289677245 +0000 UTC m=+6481.214619462" Nov 03 10:27:29.179450 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:29.179411 5015 generic.go:356] "Generic (PLEG): container finished" podID="10eabecf-4c83-4930-b92a-e478440f283d" containerID="436ecd695b07986d48ff6c4e156a24cc1f62201a812d78f8f9dfa2e79f13a222" exitCode=0 Nov 03 10:27:29.179668 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:29.179523 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" event={"ID":"10eabecf-4c83-4930-b92a-e478440f283d","Type":"ContainerDied","Data":"436ecd695b07986d48ff6c4e156a24cc1f62201a812d78f8f9dfa2e79f13a222"} Nov 03 10:27:30.335714 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:30.335688 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:30.408455 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:30.408421 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-openshift-ca\") pod \"10eabecf-4c83-4930-b92a-e478440f283d\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " Nov 03 10:27:30.408669 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:30.408534 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-xks-ca\") pod \"10eabecf-4c83-4930-b92a-e478440f283d\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " Nov 03 10:27:30.408669 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:30.408582 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10eabecf-4c83-4930-b92a-e478440f283d-kube-api-access-s4q59\" (UniqueName: \"kubernetes.io/projected/10eabecf-4c83-4930-b92a-e478440f283d-kube-api-access-s4q59\") pod \"10eabecf-4c83-4930-b92a-e478440f283d\" (UID: \"10eabecf-4c83-4930-b92a-e478440f283d\") " Nov 03 10:27:30.408878 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:30.408852 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-openshift-ca" pod "10eabecf-4c83-4930-b92a-e478440f283d" (UID: "10eabecf-4c83-4930-b92a-e478440f283d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:27:30.408999 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:30.408984 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:30.410903 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:30.410876 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-xks-ca" pod "10eabecf-4c83-4930-b92a-e478440f283d" (UID: "10eabecf-4c83-4930-b92a-e478440f283d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:27:30.411026 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:30.411005 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10eabecf-4c83-4930-b92a-e478440f283d-kube-api-access-s4q59" pod "10eabecf-4c83-4930-b92a-e478440f283d" (UID: "10eabecf-4c83-4930-b92a-e478440f283d"). InnerVolumeSpecName "kube-api-access-s4q59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:27:30.510241 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:30.510209 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10eabecf-4c83-4930-b92a-e478440f283d-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:30.510241 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:30.510238 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4q59\" (UniqueName: \"kubernetes.io/projected/10eabecf-4c83-4930-b92a-e478440f283d-kube-api-access-s4q59\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:31.193241 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:31.193211 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" Nov 03 10:27:31.193433 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:31.193243 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc" event={"ID":"10eabecf-4c83-4930-b92a-e478440f283d","Type":"ContainerDied","Data":"96dbd705dde798214ece74b69d868338651e129a7fee68729e9e22a33a53f306"} Nov 03 10:27:31.193433 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:31.193311 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96dbd705dde798214ece74b69d868338651e129a7fee68729e9e22a33a53f306" Nov 03 10:27:37.948076 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:37.948040 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h"] Nov 03 10:27:37.948723 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:37.948707 5015 cpu_manager.go:414] "containerMap: removing container" podUID="10eabecf-4c83-4930-b92a-e478440f283d" containerName="revoke-keys" Nov 03 10:27:37.948777 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:37.948728 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10eabecf-4c83-4930-b92a-e478440f283d" containerName="revoke-keys" Nov 03 10:27:37.948877 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:37.948867 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="10eabecf-4c83-4930-b92a-e478440f283d" containerName="revoke-keys" Nov 03 10:27:37.952585 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:37.952560 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:38.008799 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.008769 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h"] Nov 03 10:27:38.054544 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.054510 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:27:38.054719 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.054608 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:27:38.083774 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.083746 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsbjw\" (UniqueName: \"kubernetes.io/projected/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-kube-api-access-wsbjw\") pod \"maas-api-revoke-sub-95fa3f8f-scl9h\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:38.083926 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.083859 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95fa3f8f-scl9h\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:38.083974 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.083918 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95fa3f8f-scl9h\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:38.186162 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.186129 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbjw\" (UniqueName: \"kubernetes.io/projected/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-kube-api-access-wsbjw\") pod \"maas-api-revoke-sub-95fa3f8f-scl9h\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:38.186359 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.186206 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95fa3f8f-scl9h\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:38.186359 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.186230 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95fa3f8f-scl9h\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:38.187092 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.187072 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95fa3f8f-scl9h\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:38.189165 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.189139 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95fa3f8f-scl9h\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:38.245946 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.245910 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbjw\" (UniqueName: \"kubernetes.io/projected/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-kube-api-access-wsbjw\") pod \"maas-api-revoke-sub-95fa3f8f-scl9h\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:38.268607 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.268582 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:38.647800 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:38.647771 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h"] Nov 03 10:27:38.648165 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:27:38.648143 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ed9e13_f95d_45e0_8ec9_0a6d5a6e2b41.slice/crio-8aa4bdf11a87e339d91237ae690e722b4028aa609ec7afc74c7aec6200fe2825 WatchSource:0}: Error finding container 8aa4bdf11a87e339d91237ae690e722b4028aa609ec7afc74c7aec6200fe2825: Status 404 returned error can't find the container with id 8aa4bdf11a87e339d91237ae690e722b4028aa609ec7afc74c7aec6200fe2825 Nov 03 10:27:39.242570 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:39.242536 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" event={"ID":"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41","Type":"ContainerStarted","Data":"46c3d3b251c36f5a2875446d8708d3caaa5d584fc0182a6562b4258833f2f3d4"} Nov 03 10:27:39.242946 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:39.242586 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" event={"ID":"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41","Type":"ContainerStarted","Data":"8aa4bdf11a87e339d91237ae690e722b4028aa609ec7afc74c7aec6200fe2825"} Nov 03 10:27:39.358752 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:39.358694 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" podStartSLOduration=2.358680673 podStartE2EDuration="2.358680673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:27:39.357719746 +0000 UTC m=+6492.282661980" watchObservedRunningTime="2026-11-03 10:27:39.358680673 +0000 UTC m=+6492.283622889" Nov 03 10:27:40.250964 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:40.250924 5015 generic.go:356] "Generic (PLEG): container finished" podID="25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41" containerID="46c3d3b251c36f5a2875446d8708d3caaa5d584fc0182a6562b4258833f2f3d4" exitCode=0 Nov 03 10:27:40.251583 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:40.251014 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" event={"ID":"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41","Type":"ContainerDied","Data":"46c3d3b251c36f5a2875446d8708d3caaa5d584fc0182a6562b4258833f2f3d4"} Nov 03 10:27:41.427291 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:41.426942 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:41.522636 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:41.522526 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-openshift-ca\") pod \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " Nov 03 10:27:41.522636 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:41.522570 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-xks-ca\") pod \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " Nov 03 10:27:41.522901 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:41.522679 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-kube-api-access-wsbjw\" (UniqueName: \"kubernetes.io/projected/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-kube-api-access-wsbjw\") pod \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\" (UID: \"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41\") " Nov 03 10:27:41.522999 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:41.522959 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-openshift-ca" pod "25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41" (UID: "25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:27:41.523137 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:41.523115 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:41.525638 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:41.525612 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-kube-api-access-wsbjw" pod "25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41" (UID: "25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41"). InnerVolumeSpecName "kube-api-access-wsbjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:27:41.525796 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:41.525703 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-xks-ca" pod "25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41" (UID: "25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:27:41.630068 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:41.629965 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsbjw\" (UniqueName: \"kubernetes.io/projected/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-kube-api-access-wsbjw\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:41.630068 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:41.630011 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:42.277776 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:42.277733 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" event={"ID":"25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41","Type":"ContainerDied","Data":"8aa4bdf11a87e339d91237ae690e722b4028aa609ec7afc74c7aec6200fe2825"} Nov 03 10:27:42.278072 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:42.278050 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aa4bdf11a87e339d91237ae690e722b4028aa609ec7afc74c7aec6200fe2825" Nov 03 10:27:42.278254 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:42.278238 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h" Nov 03 10:27:46.901967 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:27:46.901930 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:27:48.707027 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.706968 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp"] Nov 03 10:27:48.708031 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.708002 5015 cpu_manager.go:414] "containerMap: removing container" podUID="25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41" containerName="revoke-keys" Nov 03 10:27:48.708031 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.708029 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41" containerName="revoke-keys" Nov 03 10:27:48.708312 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.708297 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41" containerName="revoke-keys" Nov 03 10:27:48.715350 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.712744 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:48.730637 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.730613 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60760ce0-h7shp\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:48.730773 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.730749 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60760ce0-h7shp\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:48.730857 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.730842 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x5gl\" (UniqueName: \"kubernetes.io/projected/5b816de8-c9e0-4130-863d-6463e1e3c71a-kube-api-access-5x5gl\") pod \"maas-api-revoke-sub-60760ce0-h7shp\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:48.766386 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.766346 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp"] Nov 03 10:27:48.811652 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.811615 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:27:48.811652 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.811643 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:27:48.831635 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.831606 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5gl\" (UniqueName: \"kubernetes.io/projected/5b816de8-c9e0-4130-863d-6463e1e3c71a-kube-api-access-5x5gl\") pod \"maas-api-revoke-sub-60760ce0-h7shp\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:48.831819 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.831649 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60760ce0-h7shp\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:48.831819 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.831702 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60760ce0-h7shp\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:48.832369 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.832343 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60760ce0-h7shp\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:48.834318 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.834293 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60760ce0-h7shp\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:48.890894 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:48.890864 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5gl\" (UniqueName: \"kubernetes.io/projected/5b816de8-c9e0-4130-863d-6463e1e3c71a-kube-api-access-5x5gl\") pod \"maas-api-revoke-sub-60760ce0-h7shp\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:49.028949 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:49.028917 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:49.406792 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:49.406762 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp"] Nov 03 10:27:49.407289 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:27:49.407267 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b816de8_c9e0_4130_863d_6463e1e3c71a.slice/crio-daf0615ae4335c909683962113de48d34a639b41ca1c63025a7784df99bf0025 WatchSource:0}: Error finding container daf0615ae4335c909683962113de48d34a639b41ca1c63025a7784df99bf0025: Status 404 returned error can't find the container with id daf0615ae4335c909683962113de48d34a639b41ca1c63025a7784df99bf0025 Nov 03 10:27:50.336381 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:50.336344 5015 generic.go:356] "Generic (PLEG): container finished" podID="5b816de8-c9e0-4130-863d-6463e1e3c71a" containerID="d9cc48f5bd5e04b94636ea1403b0b6903a915d1069b00262dcf5640ef2b96a3b" exitCode=0 Nov 03 10:27:50.336838 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:50.336427 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" event={"ID":"5b816de8-c9e0-4130-863d-6463e1e3c71a","Type":"ContainerDied","Data":"d9cc48f5bd5e04b94636ea1403b0b6903a915d1069b00262dcf5640ef2b96a3b"} Nov 03 10:27:50.336838 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:50.336495 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" event={"ID":"5b816de8-c9e0-4130-863d-6463e1e3c71a","Type":"ContainerStarted","Data":"daf0615ae4335c909683962113de48d34a639b41ca1c63025a7784df99bf0025"} Nov 03 10:27:51.518369 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:51.518338 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:51.561660 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:51.561606 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-xks-ca\") pod \"5b816de8-c9e0-4130-863d-6463e1e3c71a\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " Nov 03 10:27:51.561882 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:51.561680 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-openshift-ca\") pod \"5b816de8-c9e0-4130-863d-6463e1e3c71a\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " Nov 03 10:27:51.561882 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:51.561786 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b816de8-c9e0-4130-863d-6463e1e3c71a-kube-api-access-5x5gl\" (UniqueName: \"kubernetes.io/projected/5b816de8-c9e0-4130-863d-6463e1e3c71a-kube-api-access-5x5gl\") pod \"5b816de8-c9e0-4130-863d-6463e1e3c71a\" (UID: \"5b816de8-c9e0-4130-863d-6463e1e3c71a\") " Nov 03 10:27:51.562538 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:51.562504 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-openshift-ca" pod "5b816de8-c9e0-4130-863d-6463e1e3c71a" (UID: "5b816de8-c9e0-4130-863d-6463e1e3c71a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:27:51.563896 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:51.563870 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-xks-ca" pod "5b816de8-c9e0-4130-863d-6463e1e3c71a" (UID: "5b816de8-c9e0-4130-863d-6463e1e3c71a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:27:51.564546 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:51.564521 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b816de8-c9e0-4130-863d-6463e1e3c71a-kube-api-access-5x5gl" pod "5b816de8-c9e0-4130-863d-6463e1e3c71a" (UID: "5b816de8-c9e0-4130-863d-6463e1e3c71a"). InnerVolumeSpecName "kube-api-access-5x5gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:27:51.663300 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:51.663209 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:51.663300 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:51.663242 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b816de8-c9e0-4130-863d-6463e1e3c71a-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:51.663300 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:51.663257 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x5gl\" (UniqueName: \"kubernetes.io/projected/5b816de8-c9e0-4130-863d-6463e1e3c71a-kube-api-access-5x5gl\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:27:52.355185 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:52.355149 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" event={"ID":"5b816de8-c9e0-4130-863d-6463e1e3c71a","Type":"ContainerDied","Data":"daf0615ae4335c909683962113de48d34a639b41ca1c63025a7784df99bf0025"} Nov 03 10:27:52.355504 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:52.355460 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daf0615ae4335c909683962113de48d34a639b41ca1c63025a7784df99bf0025" Nov 03 10:27:52.355706 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:52.355693 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp" Nov 03 10:27:54.070502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:54.069797 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk"] Nov 03 10:27:54.115286 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:54.115254 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb20ca37-pxdwk"] Nov 03 10:27:55.204346 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:55.204307 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2" path="/var/lib/kubelet/pods/3a8a4e0f-f857-4d07-9bc0-c4a39ac79df2/volumes" Nov 03 10:27:59.515369 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.515336 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln"] Nov 03 10:27:59.515821 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.515808 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5b816de8-c9e0-4130-863d-6463e1e3c71a" containerName="revoke-keys" Nov 03 10:27:59.515868 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.515823 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b816de8-c9e0-4130-863d-6463e1e3c71a" containerName="revoke-keys" Nov 03 10:27:59.515915 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.515906 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b816de8-c9e0-4130-863d-6463e1e3c71a" containerName="revoke-keys" Nov 03 10:27:59.520194 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.520178 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:27:59.577526 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.577496 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln"] Nov 03 10:27:59.588095 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.588064 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686b0d6f-dstln\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:27:59.588236 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.588202 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfsz9\" (UniqueName: \"kubernetes.io/projected/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-kube-api-access-pfsz9\") pod \"maas-api-revoke-sub-686b0d6f-dstln\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:27:59.588370 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.588349 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686b0d6f-dstln\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:27:59.623222 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.623192 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:27:59.623358 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.623269 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:27:59.689760 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.689729 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686b0d6f-dstln\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:27:59.689904 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.689771 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686b0d6f-dstln\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:27:59.689904 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.689815 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfsz9\" (UniqueName: \"kubernetes.io/projected/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-kube-api-access-pfsz9\") pod \"maas-api-revoke-sub-686b0d6f-dstln\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:27:59.690529 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.690468 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686b0d6f-dstln\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:27:59.692237 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.692218 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686b0d6f-dstln\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:27:59.749550 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.749526 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfsz9\" (UniqueName: \"kubernetes.io/projected/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-kube-api-access-pfsz9\") pod \"maas-api-revoke-sub-686b0d6f-dstln\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:27:59.837077 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:27:59.836987 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:28:00.217300 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:00.217270 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln"] Nov 03 10:28:00.217991 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:28:00.217958 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702136bd_30d2_41bd_a7f4_d6fdcfc47d66.slice/crio-c291059f63d905bb442152ef29418e08e08664814e45c529e13dc586518d265e WatchSource:0}: Error finding container c291059f63d905bb442152ef29418e08e08664814e45c529e13dc586518d265e: Status 404 returned error can't find the container with id c291059f63d905bb442152ef29418e08e08664814e45c529e13dc586518d265e Nov 03 10:28:00.452619 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:00.452588 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" event={"ID":"702136bd-30d2-41bd-a7f4-d6fdcfc47d66","Type":"ContainerStarted","Data":"c291059f63d905bb442152ef29418e08e08664814e45c529e13dc586518d265e"} Nov 03 10:28:01.460671 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:01.460573 5015 generic.go:356] "Generic (PLEG): container finished" podID="702136bd-30d2-41bd-a7f4-d6fdcfc47d66" containerID="11dc7381a2da4e5d2557c899415b9fd1e595d3d19b93bf97ded42c926dbf4f96" exitCode=0 Nov 03 10:28:01.461123 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:01.460664 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" event={"ID":"702136bd-30d2-41bd-a7f4-d6fdcfc47d66","Type":"ContainerDied","Data":"11dc7381a2da4e5d2557c899415b9fd1e595d3d19b93bf97ded42c926dbf4f96"} Nov 03 10:28:02.654294 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:02.654258 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:28:02.723045 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:02.723016 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-openshift-ca\") pod \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " Nov 03 10:28:02.723265 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:02.723066 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-kube-api-access-pfsz9\" (UniqueName: \"kubernetes.io/projected/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-kube-api-access-pfsz9\") pod \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " Nov 03 10:28:02.723265 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:02.723158 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-xks-ca\") pod \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\" (UID: \"702136bd-30d2-41bd-a7f4-d6fdcfc47d66\") " Nov 03 10:28:02.723798 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:02.723444 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-openshift-ca" pod "702136bd-30d2-41bd-a7f4-d6fdcfc47d66" (UID: "702136bd-30d2-41bd-a7f4-d6fdcfc47d66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:28:02.723798 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:02.723608 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:28:02.726296 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:02.726268 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-xks-ca" pod "702136bd-30d2-41bd-a7f4-d6fdcfc47d66" (UID: "702136bd-30d2-41bd-a7f4-d6fdcfc47d66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:28:02.726398 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:02.726378 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-kube-api-access-pfsz9" pod "702136bd-30d2-41bd-a7f4-d6fdcfc47d66" (UID: "702136bd-30d2-41bd-a7f4-d6fdcfc47d66"). InnerVolumeSpecName "kube-api-access-pfsz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:28:02.824494 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:02.824442 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfsz9\" (UniqueName: \"kubernetes.io/projected/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-kube-api-access-pfsz9\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:28:02.824674 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:02.824504 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702136bd-30d2-41bd-a7f4-d6fdcfc47d66-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:28:03.501775 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:03.501738 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" event={"ID":"702136bd-30d2-41bd-a7f4-d6fdcfc47d66","Type":"ContainerDied","Data":"c291059f63d905bb442152ef29418e08e08664814e45c529e13dc586518d265e"} Nov 03 10:28:03.502079 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:03.502056 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c291059f63d905bb442152ef29418e08e08664814e45c529e13dc586518d265e" Nov 03 10:28:03.502348 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:03.502331 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln" Nov 03 10:28:05.052327 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:05.052291 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq"] Nov 03 10:28:05.102771 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:05.102737 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770a93b5-zj8jq"] Nov 03 10:28:05.204425 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:05.204346 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee675d0-0851-4019-b55f-124f30eb3c7a" path="/var/lib/kubelet/pods/8ee675d0-0851-4019-b55f-124f30eb3c7a/volumes" Nov 03 10:28:10.313751 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.313719 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t"] Nov 03 10:28:10.314265 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.314252 5015 cpu_manager.go:414] "containerMap: removing container" podUID="702136bd-30d2-41bd-a7f4-d6fdcfc47d66" containerName="revoke-keys" Nov 03 10:28:10.314321 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.314266 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702136bd-30d2-41bd-a7f4-d6fdcfc47d66" containerName="revoke-keys" Nov 03 10:28:10.314367 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.314357 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="702136bd-30d2-41bd-a7f4-d6fdcfc47d66" containerName="revoke-keys" Nov 03 10:28:10.319112 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.319088 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:10.373633 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.373607 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t"] Nov 03 10:28:10.420268 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.420236 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:28:10.420268 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.420267 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:28:10.452091 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.452064 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f53f53c1-tlp9t\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:10.452247 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.452100 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72q99\" (UniqueName: \"kubernetes.io/projected/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-kube-api-access-72q99\") pod \"maas-api-revoke-sub-f53f53c1-tlp9t\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:10.452247 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.452173 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f53f53c1-tlp9t\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:10.553758 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.553718 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f53f53c1-tlp9t\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:10.553962 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.553852 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f53f53c1-tlp9t\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:10.553962 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.553887 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72q99\" (UniqueName: \"kubernetes.io/projected/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-kube-api-access-72q99\") pod \"maas-api-revoke-sub-f53f53c1-tlp9t\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:10.554469 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.554450 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f53f53c1-tlp9t\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:10.556364 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.556341 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f53f53c1-tlp9t\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:10.613011 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.612942 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72q99\" (UniqueName: \"kubernetes.io/projected/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-kube-api-access-72q99\") pod \"maas-api-revoke-sub-f53f53c1-tlp9t\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:10.632923 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:10.632899 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:11.014354 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:11.014313 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t"] Nov 03 10:28:11.015393 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:28:11.015361 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6355a158_af38_4f4c_8ae7_afa79ee6ce1b.slice/crio-e1091d8ae84af5a08b0caf9dcccc1ae28f7a57d890ed401f4a0d6f559912b0f0 WatchSource:0}: Error finding container e1091d8ae84af5a08b0caf9dcccc1ae28f7a57d890ed401f4a0d6f559912b0f0: Status 404 returned error can't find the container with id e1091d8ae84af5a08b0caf9dcccc1ae28f7a57d890ed401f4a0d6f559912b0f0 Nov 03 10:28:11.563542 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:11.563504 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" event={"ID":"6355a158-af38-4f4c-8ae7-afa79ee6ce1b","Type":"ContainerStarted","Data":"8f6e18ea33bb44aa8c6a9d78e71f2141c45344155fe0a7ec72a401d34db0f5a2"} Nov 03 10:28:11.563939 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:11.563572 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" event={"ID":"6355a158-af38-4f4c-8ae7-afa79ee6ce1b","Type":"ContainerStarted","Data":"e1091d8ae84af5a08b0caf9dcccc1ae28f7a57d890ed401f4a0d6f559912b0f0"} Nov 03 10:28:11.680716 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:11.680611 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" podStartSLOduration=1.680596937 podStartE2EDuration="1.680596937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:11.679044274 +0000 UTC m=+6524.603986511" watchObservedRunningTime="2026-11-03 10:28:11.680596937 +0000 UTC m=+6524.605539153" Nov 03 10:28:12.579463 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:12.579428 5015 generic.go:356] "Generic (PLEG): container finished" podID="6355a158-af38-4f4c-8ae7-afa79ee6ce1b" containerID="8f6e18ea33bb44aa8c6a9d78e71f2141c45344155fe0a7ec72a401d34db0f5a2" exitCode=0 Nov 03 10:28:12.580000 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:12.579558 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" event={"ID":"6355a158-af38-4f4c-8ae7-afa79ee6ce1b","Type":"ContainerDied","Data":"8f6e18ea33bb44aa8c6a9d78e71f2141c45344155fe0a7ec72a401d34db0f5a2"} Nov 03 10:28:13.755168 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:13.755133 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:13.787801 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:13.787760 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-openshift-ca\") pod \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " Nov 03 10:28:13.788028 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:13.787870 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-kube-api-access-72q99\" (UniqueName: \"kubernetes.io/projected/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-kube-api-access-72q99\") pod \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " Nov 03 10:28:13.788028 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:13.787908 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-xks-ca\") pod \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\" (UID: \"6355a158-af38-4f4c-8ae7-afa79ee6ce1b\") " Nov 03 10:28:13.788203 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:13.788171 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-openshift-ca" pod "6355a158-af38-4f4c-8ae7-afa79ee6ce1b" (UID: "6355a158-af38-4f4c-8ae7-afa79ee6ce1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:28:13.788320 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:13.788299 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:28:13.790255 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:13.790226 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-kube-api-access-72q99" pod "6355a158-af38-4f4c-8ae7-afa79ee6ce1b" (UID: "6355a158-af38-4f4c-8ae7-afa79ee6ce1b"). InnerVolumeSpecName "kube-api-access-72q99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:28:13.790715 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:13.790690 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-xks-ca" pod "6355a158-af38-4f4c-8ae7-afa79ee6ce1b" (UID: "6355a158-af38-4f4c-8ae7-afa79ee6ce1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:28:13.911229 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:13.911132 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72q99\" (UniqueName: \"kubernetes.io/projected/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-kube-api-access-72q99\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:28:13.911470 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:13.911448 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6355a158-af38-4f4c-8ae7-afa79ee6ce1b-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:28:14.619501 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:14.619066 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" event={"ID":"6355a158-af38-4f4c-8ae7-afa79ee6ce1b","Type":"ContainerDied","Data":"e1091d8ae84af5a08b0caf9dcccc1ae28f7a57d890ed401f4a0d6f559912b0f0"} Nov 03 10:28:14.619501 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:14.619134 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1091d8ae84af5a08b0caf9dcccc1ae28f7a57d890ed401f4a0d6f559912b0f0" Nov 03 10:28:14.619501 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:14.619202 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t" Nov 03 10:28:15.528114 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:15.528072 5015 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:28:15.528832 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:15.528787 5015 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:28:16.901133 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:28:16.901100 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:28:21.075708 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.075674 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd"] Nov 03 10:28:21.076244 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.076152 5015 cpu_manager.go:414] "containerMap: removing container" podUID="6355a158-af38-4f4c-8ae7-afa79ee6ce1b" containerName="revoke-keys" Nov 03 10:28:21.076244 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.076165 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6355a158-af38-4f4c-8ae7-afa79ee6ce1b" containerName="revoke-keys" Nov 03 10:28:21.076244 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.076243 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="6355a158-af38-4f4c-8ae7-afa79ee6ce1b" containerName="revoke-keys" Nov 03 10:28:21.079707 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.079683 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:21.136423 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.136394 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd"] Nov 03 10:28:21.182381 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.182356 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 10:28:21.182563 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.182415 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:28:21.185868 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.185836 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbj9\" (UniqueName: \"kubernetes.io/projected/5f4b3383-4c1c-4de5-94ea-43aa01896169-kube-api-access-zgbj9\") pod \"maas-api-revoke-sub-00a990b0-ssctd\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:21.185996 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.185879 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00a990b0-ssctd\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:21.185996 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.185900 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00a990b0-ssctd\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:21.287172 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.287136 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbj9\" (UniqueName: \"kubernetes.io/projected/5f4b3383-4c1c-4de5-94ea-43aa01896169-kube-api-access-zgbj9\") pod \"maas-api-revoke-sub-00a990b0-ssctd\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:21.287361 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.287181 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00a990b0-ssctd\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:21.287361 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.287201 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00a990b0-ssctd\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:21.287837 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.287806 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00a990b0-ssctd\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:21.289787 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.289767 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00a990b0-ssctd\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:21.346956 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.346897 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbj9\" (UniqueName: \"kubernetes.io/projected/5f4b3383-4c1c-4de5-94ea-43aa01896169-kube-api-access-zgbj9\") pod \"maas-api-revoke-sub-00a990b0-ssctd\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:21.393987 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.393963 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:21.777291 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:21.777259 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd"] Nov 03 10:28:21.777924 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:28:21.777900 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4b3383_4c1c_4de5_94ea_43aa01896169.slice/crio-2a19f276c30beac3b6262b0b22ce98ebf97a8a547c206dabd7af140ef9e981a8 WatchSource:0}: Error finding container 2a19f276c30beac3b6262b0b22ce98ebf97a8a547c206dabd7af140ef9e981a8: Status 404 returned error can't find the container with id 2a19f276c30beac3b6262b0b22ce98ebf97a8a547c206dabd7af140ef9e981a8 Nov 03 10:28:22.673813 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:22.673781 5015 generic.go:356] "Generic (PLEG): container finished" podID="5f4b3383-4c1c-4de5-94ea-43aa01896169" containerID="10b77b15891cff5ec321f1081966e7caa3e4bf75d373e35323258632285f34c3" exitCode=0 Nov 03 10:28:22.674159 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:22.673816 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" event={"ID":"5f4b3383-4c1c-4de5-94ea-43aa01896169","Type":"ContainerDied","Data":"10b77b15891cff5ec321f1081966e7caa3e4bf75d373e35323258632285f34c3"} Nov 03 10:28:22.674159 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:22.673871 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" event={"ID":"5f4b3383-4c1c-4de5-94ea-43aa01896169","Type":"ContainerStarted","Data":"2a19f276c30beac3b6262b0b22ce98ebf97a8a547c206dabd7af140ef9e981a8"} Nov 03 10:28:23.854818 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:23.854790 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:23.914129 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:23.912830 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-xks-ca\") pod \"5f4b3383-4c1c-4de5-94ea-43aa01896169\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " Nov 03 10:28:23.914129 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:23.912888 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-openshift-ca\") pod \"5f4b3383-4c1c-4de5-94ea-43aa01896169\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " Nov 03 10:28:23.914129 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:23.912932 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4b3383-4c1c-4de5-94ea-43aa01896169-kube-api-access-zgbj9\" (UniqueName: \"kubernetes.io/projected/5f4b3383-4c1c-4de5-94ea-43aa01896169-kube-api-access-zgbj9\") pod \"5f4b3383-4c1c-4de5-94ea-43aa01896169\" (UID: \"5f4b3383-4c1c-4de5-94ea-43aa01896169\") " Nov 03 10:28:23.914592 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:23.914566 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-openshift-ca" pod "5f4b3383-4c1c-4de5-94ea-43aa01896169" (UID: "5f4b3383-4c1c-4de5-94ea-43aa01896169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:28:23.917283 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:23.917258 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-xks-ca" pod "5f4b3383-4c1c-4de5-94ea-43aa01896169" (UID: "5f4b3383-4c1c-4de5-94ea-43aa01896169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:28:23.917640 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:23.917605 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4b3383-4c1c-4de5-94ea-43aa01896169-kube-api-access-zgbj9" pod "5f4b3383-4c1c-4de5-94ea-43aa01896169" (UID: "5f4b3383-4c1c-4de5-94ea-43aa01896169"). InnerVolumeSpecName "kube-api-access-zgbj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:28:24.018250 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:24.017144 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-xks-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:28:24.018250 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:24.017185 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4b3383-4c1c-4de5-94ea-43aa01896169-maas-api-openshift-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:28:24.018250 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:24.017217 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgbj9\" (UniqueName: \"kubernetes.io/projected/5f4b3383-4c1c-4de5-94ea-43aa01896169-kube-api-access-zgbj9\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:28:24.724026 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:24.723997 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" Nov 03 10:28:24.724026 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:24.724013 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd" event={"ID":"5f4b3383-4c1c-4de5-94ea-43aa01896169","Type":"ContainerDied","Data":"2a19f276c30beac3b6262b0b22ce98ebf97a8a547c206dabd7af140ef9e981a8"} Nov 03 10:28:24.724328 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:24.724066 5015 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a19f276c30beac3b6262b0b22ce98ebf97a8a547c206dabd7af140ef9e981a8" Nov 03 10:28:30.325938 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:30.325903 5015 scope.go:122] "RemoveContainer" containerID="819d587e6b98d9b95f15bd3eae8fb393f5020b13ade28e17cd69dd36971209bf" Nov 03 10:28:30.336063 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:30.336043 5015 scope.go:122] "RemoveContainer" containerID="7746b7590b27c2a7d0d3947a356c8f9e74bdfb4142183687a69bdbb6840de23e" Nov 03 10:28:40.053587 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:40.053549 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf"] Nov 03 10:28:40.106256 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:40.106217 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5be9a74-fcdpf"] Nov 03 10:28:41.204248 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:41.204208 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9204425e-8e3c-4f97-8670-5d6532a03204" path="/var/lib/kubelet/pods/9204425e-8e3c-4f97-8670-5d6532a03204/volumes" Nov 03 10:28:46.914507 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:28:46.914446 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:28:48.411527 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:28:48.411499 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dfcc7c588-7hvqz_ae051d13-de71-4752-b918-8bb60b21002d/authorino/0.log" Nov 03 10:29:16.901349 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:29:16.901309 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:29:24.053657 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:24.053625 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d"] Nov 03 10:29:24.103843 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:24.103802 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587ba938-sgt9d"] Nov 03 10:29:25.204274 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:25.204244 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="968346cd-8243-4b92-94da-bd610905fad5" path="/var/lib/kubelet/pods/968346cd-8243-4b92-94da-bd610905fad5/volumes" Nov 03 10:29:28.406449 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:28.406422 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:29:28.433932 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:28.433909 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:29:30.364911 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:30.364878 5015 scope.go:122] "RemoveContainer" containerID="4f11233e5b461fcf00a5dd21f2677c54977a92211e8f6a74a1b7d8da1c3fff18" Nov 03 10:29:30.374788 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:30.374769 5015 scope.go:122] "RemoveContainer" containerID="4756c39073983164f3b4a3653abba800b8667089e450759824c3ac71fa01e410" Nov 03 10:29:30.632902 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:30.632806 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wwbxd_e6a48fda-d04e-423f-89dc-a640c4b2d513/manager/0.log" Nov 03 10:29:34.081793 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:34.081761 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-00a990b0-ssctd_5f4b3383-4c1c-4de5-94ea-43aa01896169/revoke-keys/0.log" Nov 03 10:29:34.282650 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:34.282607 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-06b0b816-sfmjc_10eabecf-4c83-4930-b92a-e478440f283d/revoke-keys/0.log" Nov 03 10:29:34.495971 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:34.495934 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-11318af9-rnqdq_249c8c10-8e03-41c6-997a-c413fff553e0/revoke-keys/0.log" Nov 03 10:29:34.705276 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:34.705245 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-189eb33c-hj2zm_355aac11-afb8-4865-a7f1-50db3ec3d7d2/revoke-keys/0.log" Nov 03 10:29:34.906541 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:34.906449 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1c9aafc4-xp7hh_cdc612f6-eb87-424c-ad70-07b5581e27a4/revoke-keys/0.log" Nov 03 10:29:35.053269 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:35.053237 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7"] Nov 03 10:29:35.102839 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:35.102799 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d5f895-l5dd7"] Nov 03 10:29:35.115199 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:35.115168 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3028a06a-299zt_87493051-b17d-4aaa-8674-dbf6575b059d/revoke-keys/0.log" Nov 03 10:29:35.203470 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:35.203375 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a95d0313-2000-4c56-a7fa-d4d4a30e23cb" path="/var/lib/kubelet/pods/a95d0313-2000-4c56-a7fa-d4d4a30e23cb/volumes" Nov 03 10:29:35.337628 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:35.337598 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-60760ce0-h7shp_5b816de8-c9e0-4130-863d-6463e1e3c71a/revoke-keys/0.log" Nov 03 10:29:35.560407 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:35.560379 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-686b0d6f-dstln_702136bd-30d2-41bd-a7f4-d6fdcfc47d66/revoke-keys/0.log" Nov 03 10:29:35.779505 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:35.779438 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-861756ce-tr49r_487e8c78-f695-4dcf-aa4e-b8668a69c646/revoke-keys/0.log" Nov 03 10:29:35.999231 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:35.999202 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-95fa3f8f-scl9h_25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41/revoke-keys/0.log" Nov 03 10:29:36.380980 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:36.380889 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c5557744-rr2ks_8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a/revoke-keys/0.log" Nov 03 10:29:36.595015 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:36.594987 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d0a4d67c-bcn26_85404c28-3ba7-45f8-a61b-6e726be726e9/revoke-keys/0.log" Nov 03 10:29:36.810811 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:36.810774 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d0fd88da-c4bml_72de1663-8c9b-4c9b-b186-8e2872f1622b/revoke-keys/0.log" Nov 03 10:29:37.026692 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:37.026653 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f53f53c1-tlp9t_6355a158-af38-4f4c-8ae7-afa79ee6ce1b/revoke-keys/0.log" Nov 03 10:29:37.237657 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:37.237627 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-lf65p_4757c90e-41ba-46ed-8388-7f8c489d4b86/postgres/0.log" Nov 03 10:29:46.901424 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:29:46.901377 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:29:49.168207 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:49.168171 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dfcc7c588-7hvqz_ae051d13-de71-4752-b918-8bb60b21002d/authorino/0.log" Nov 03 10:29:49.582727 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:49.582697 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rc42m_1786e9e7-2202-4fce-aaf4-2f0b30932688/manager/0.log" Nov 03 10:29:50.483202 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:50.483172 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cr2kg_19f7d098-9626-4724-a15e-85253b3dfd54/manager/0.log" Nov 03 10:29:52.311901 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:52.311872 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-86rlh_af4db30f-c036-4273-b29e-733cd2dbe65f/istio-proxy/0.log" Nov 03 10:29:54.482553 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:54.482520 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-dhjmc_ded8f558-e327-4178-a2ac-948e7f005ef1/istio-proxy/0.log" Nov 03 10:29:55.691716 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:55.691684 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lg5gm_0d1bcdca-f1fe-45db-98df-66f3ebd5696e/istio-proxy/0.log" Nov 03 10:29:56.011338 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:56.011303 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lkg9f_fdb2f094-67e8-438b-80b8-c1ef4521b355/istio-proxy/0.log" Nov 03 10:29:56.287861 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:56.287785 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7_f9d5e9a1-bee4-4750-90bf-892c58ff29a5/istio-proxy/0.log" Nov 03 10:29:56.504533 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:56.504491 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cf9dd8f69-nc6dw_6d2b9c62-2c04-43b1-a529-cfb16c231ecd/payload-pre-processing/0.log" Nov 03 10:29:56.726964 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:56.726927 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85f55897bb-jwz7j_8b000dd7-72c1-491d-8500-6acde8c672b0/payload-processing/0.log" Nov 03 10:29:56.949388 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:56.949355 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b64c9d49-xhzz8_70bfc17c-8e18-473a-a8fe-e386112cfaa4/router/0.log" Nov 03 10:29:58.459145 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:58.459112 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-qrhwd_6f925e33-6569-4ac4-9422-1ac38473de7a/main/0.log" Nov 03 10:29:58.935813 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:58.935786 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8_12c4a382-3942-493e-84a4-85d061fc87e4/main/0.log" Nov 03 10:29:59.412132 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:59.412097 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9884s_4537f2eb-9880-4ba9-a64a-0c25e21521dd/main/0.log" Nov 03 10:29:59.687606 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:29:59.687532 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6_21ad0cfe-e3c6-4770-b0b3-7c512e7f1830/main/0.log" Nov 03 10:30:00.172585 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.172552 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2"] Nov 03 10:30:00.173844 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.173817 5015 cpu_manager.go:414] "containerMap: removing container" podUID="5f4b3383-4c1c-4de5-94ea-43aa01896169" containerName="revoke-keys" Nov 03 10:30:00.174015 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.174002 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4b3383-4c1c-4de5-94ea-43aa01896169" containerName="revoke-keys" Nov 03 10:30:00.175236 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.175211 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4b3383-4c1c-4de5-94ea-43aa01896169" containerName="revoke-keys" Nov 03 10:30:00.180195 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.180166 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" Nov 03 10:30:00.235135 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.235099 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2"] Nov 03 10:30:00.278764 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.278731 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70de685-f6c7-4ae8-8b67-623d11b5e83b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895030-gzjs2\" (UID: \"a70de685-f6c7-4ae8-8b67-623d11b5e83b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" Nov 03 10:30:00.278913 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.278776 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw6mz\" (UniqueName: \"kubernetes.io/projected/a70de685-f6c7-4ae8-8b67-623d11b5e83b-kube-api-access-bw6mz\") pod \"maas-api-key-cleanup-29895030-gzjs2\" (UID: \"a70de685-f6c7-4ae8-8b67-623d11b5e83b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" Nov 03 10:30:00.282103 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.282082 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\"" Nov 03 10:30:00.379375 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.379347 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70de685-f6c7-4ae8-8b67-623d11b5e83b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895030-gzjs2\" (UID: \"a70de685-f6c7-4ae8-8b67-623d11b5e83b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" Nov 03 10:30:00.379572 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.379387 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw6mz\" (UniqueName: \"kubernetes.io/projected/a70de685-f6c7-4ae8-8b67-623d11b5e83b-kube-api-access-bw6mz\") pod \"maas-api-key-cleanup-29895030-gzjs2\" (UID: \"a70de685-f6c7-4ae8-8b67-623d11b5e83b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" Nov 03 10:30:00.379945 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.379924 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70de685-f6c7-4ae8-8b67-623d11b5e83b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895030-gzjs2\" (UID: \"a70de685-f6c7-4ae8-8b67-623d11b5e83b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" Nov 03 10:30:00.439192 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.439116 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw6mz\" (UniqueName: \"kubernetes.io/projected/a70de685-f6c7-4ae8-8b67-623d11b5e83b-kube-api-access-bw6mz\") pod \"maas-api-key-cleanup-29895030-gzjs2\" (UID: \"a70de685-f6c7-4ae8-8b67-623d11b5e83b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" Nov 03 10:30:00.493987 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.493964 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" Nov 03 10:30:00.869702 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:00.869675 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2"] Nov 03 10:30:00.871108 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:30:00.871072 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70de685_f6c7_4ae8_8b67_623d11b5e83b.slice/crio-72ea1eb0b6016713905cac4cab6e16367259ec1948a7fa5573377de8b4e5c682 WatchSource:0}: Error finding container 72ea1eb0b6016713905cac4cab6e16367259ec1948a7fa5573377de8b4e5c682: Status 404 returned error can't find the container with id 72ea1eb0b6016713905cac4cab6e16367259ec1948a7fa5573377de8b4e5c682 Nov 03 10:30:01.427823 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:01.427738 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" event={"ID":"a70de685-f6c7-4ae8-8b67-623d11b5e83b","Type":"ContainerStarted","Data":"39e23b691fe2a58a5794383aa5fdde0e59eb6dfe382a36de12b723f3eb399343"} Nov 03 10:30:01.427823 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:01.427800 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" event={"ID":"a70de685-f6c7-4ae8-8b67-623d11b5e83b","Type":"ContainerStarted","Data":"72ea1eb0b6016713905cac4cab6e16367259ec1948a7fa5573377de8b4e5c682"} Nov 03 10:30:01.547088 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:01.547037 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" podStartSLOduration=1.547024932 podStartE2EDuration="1.547024932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:30:01.543416701 +0000 UTC m=+6634.468358940" watchObservedRunningTime="2026-11-03 10:30:01.547024932 +0000 UTC m=+6634.471967151" Nov 03 10:30:02.436553 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:02.436515 5015 generic.go:356] "Generic (PLEG): container finished" podID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" containerID="39e23b691fe2a58a5794383aa5fdde0e59eb6dfe382a36de12b723f3eb399343" exitCode=60 Nov 03 10:30:02.437068 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:02.436645 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" event={"ID":"a70de685-f6c7-4ae8-8b67-623d11b5e83b","Type":"ContainerDied","Data":"39e23b691fe2a58a5794383aa5fdde0e59eb6dfe382a36de12b723f3eb399343"} Nov 03 10:30:02.437068 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:02.436875 5015 scope.go:122] "RemoveContainer" containerID="39e23b691fe2a58a5794383aa5fdde0e59eb6dfe382a36de12b723f3eb399343" Nov 03 10:30:03.445288 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:03.445250 5015 generic.go:356] "Generic (PLEG): container finished" podID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" containerID="02e1bbd8ba5d428ff919a9d3670a53b14ad5582f4de309ff814a57358479d97b" exitCode=60 Nov 03 10:30:03.445812 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:03.445342 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" event={"ID":"a70de685-f6c7-4ae8-8b67-623d11b5e83b","Type":"ContainerDied","Data":"02e1bbd8ba5d428ff919a9d3670a53b14ad5582f4de309ff814a57358479d97b"} Nov 03 10:30:03.445812 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:03.445412 5015 scope.go:122] "RemoveContainer" containerID="39e23b691fe2a58a5794383aa5fdde0e59eb6dfe382a36de12b723f3eb399343" Nov 03 10:30:03.445812 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:03.445644 5015 scope.go:122] "RemoveContainer" containerID="02e1bbd8ba5d428ff919a9d3670a53b14ad5582f4de309ff814a57358479d97b" Nov 03 10:30:03.445959 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:30:03.445919 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895030-gzjs2_odh-ai-gateway-infra(a70de685-f6c7-4ae8-8b67-623d11b5e83b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" podUID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" Nov 03 10:30:04.453557 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:04.453533 5015 scope.go:122] "RemoveContainer" containerID="02e1bbd8ba5d428ff919a9d3670a53b14ad5582f4de309ff814a57358479d97b" Nov 03 10:30:04.453959 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:30:04.453758 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895030-gzjs2_odh-ai-gateway-infra(a70de685-f6c7-4ae8-8b67-623d11b5e83b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" podUID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" Nov 03 10:30:09.784937 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:09.784903 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5d4t6/must-gather-tm2kj"] Nov 03 10:30:09.790243 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:09.790223 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5d4t6/must-gather-tm2kj" Nov 03 10:30:09.845993 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:09.845962 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5d4t6/must-gather-tm2kj"] Nov 03 10:30:09.879250 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:09.879224 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92pt\" (UniqueName: \"kubernetes.io/projected/61d5bca4-1b98-4866-b0c0-f0bad41a5e7f-kube-api-access-j92pt\") pod \"must-gather-tm2kj\" (UID: \"61d5bca4-1b98-4866-b0c0-f0bad41a5e7f\") " pod="openshift-must-gather-5d4t6/must-gather-tm2kj" Nov 03 10:30:09.879378 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:09.879282 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/61d5bca4-1b98-4866-b0c0-f0bad41a5e7f-must-gather-output\") pod \"must-gather-tm2kj\" (UID: \"61d5bca4-1b98-4866-b0c0-f0bad41a5e7f\") " pod="openshift-must-gather-5d4t6/must-gather-tm2kj" Nov 03 10:30:09.893091 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:09.893069 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5d4t6\"/\"openshift-service-ca.crt\"" Nov 03 10:30:09.893091 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:09.893079 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5d4t6\"/\"default-dockercfg-k4tff\"" Nov 03 10:30:09.893270 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:09.893115 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5d4t6\"/\"kube-root-ca.crt\"" Nov 03 10:30:09.980254 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:09.980212 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92pt\" (UniqueName: \"kubernetes.io/projected/61d5bca4-1b98-4866-b0c0-f0bad41a5e7f-kube-api-access-j92pt\") pod \"must-gather-tm2kj\" (UID: \"61d5bca4-1b98-4866-b0c0-f0bad41a5e7f\") " pod="openshift-must-gather-5d4t6/must-gather-tm2kj" Nov 03 10:30:09.980430 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:09.980277 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/61d5bca4-1b98-4866-b0c0-f0bad41a5e7f-must-gather-output\") pod \"must-gather-tm2kj\" (UID: \"61d5bca4-1b98-4866-b0c0-f0bad41a5e7f\") " pod="openshift-must-gather-5d4t6/must-gather-tm2kj" Nov 03 10:30:09.980682 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:09.980663 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/61d5bca4-1b98-4866-b0c0-f0bad41a5e7f-must-gather-output\") pod \"must-gather-tm2kj\" (UID: \"61d5bca4-1b98-4866-b0c0-f0bad41a5e7f\") " pod="openshift-must-gather-5d4t6/must-gather-tm2kj" Nov 03 10:30:10.039568 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:10.039508 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92pt\" (UniqueName: \"kubernetes.io/projected/61d5bca4-1b98-4866-b0c0-f0bad41a5e7f-kube-api-access-j92pt\") pod \"must-gather-tm2kj\" (UID: \"61d5bca4-1b98-4866-b0c0-f0bad41a5e7f\") " pod="openshift-must-gather-5d4t6/must-gather-tm2kj" Nov 03 10:30:10.104659 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:10.104628 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5d4t6/must-gather-tm2kj" Nov 03 10:30:10.487072 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:10.487035 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5d4t6/must-gather-tm2kj"] Nov 03 10:30:10.487564 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:30:10.487527 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61d5bca4_1b98_4866_b0c0_f0bad41a5e7f.slice/crio-37639049225fb994b5d3384de3f6e839fad7388ae7b61cb54211f7aebb2bbce5 WatchSource:0}: Error finding container 37639049225fb994b5d3384de3f6e839fad7388ae7b61cb54211f7aebb2bbce5: Status 404 returned error can't find the container with id 37639049225fb994b5d3384de3f6e839fad7388ae7b61cb54211f7aebb2bbce5 Nov 03 10:30:10.489292 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:10.489277 5015 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 10:30:11.500750 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:11.500716 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5d4t6/must-gather-tm2kj" event={"ID":"61d5bca4-1b98-4866-b0c0-f0bad41a5e7f","Type":"ContainerStarted","Data":"37639049225fb994b5d3384de3f6e839fad7388ae7b61cb54211f7aebb2bbce5"} Nov 03 10:30:14.200169 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:14.199742 5015 scope.go:122] "RemoveContainer" containerID="02e1bbd8ba5d428ff919a9d3670a53b14ad5582f4de309ff814a57358479d97b" Nov 03 10:30:15.535548 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:15.535442 5015 generic.go:356] "Generic (PLEG): container finished" podID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" containerID="e4eed314cd2da1ff88969af74b5247c13f9d3a310c79ea911481fef7778fdaf1" exitCode=60 Nov 03 10:30:15.535548 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:15.535518 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" event={"ID":"a70de685-f6c7-4ae8-8b67-623d11b5e83b","Type":"ContainerDied","Data":"e4eed314cd2da1ff88969af74b5247c13f9d3a310c79ea911481fef7778fdaf1"} Nov 03 10:30:15.536078 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:15.535605 5015 scope.go:122] "RemoveContainer" containerID="02e1bbd8ba5d428ff919a9d3670a53b14ad5582f4de309ff814a57358479d97b" Nov 03 10:30:15.536078 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:15.535991 5015 scope.go:122] "RemoveContainer" containerID="e4eed314cd2da1ff88969af74b5247c13f9d3a310c79ea911481fef7778fdaf1" Nov 03 10:30:15.536276 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:30:15.536254 5015 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895030-gzjs2_odh-ai-gateway-infra(a70de685-f6c7-4ae8-8b67-623d11b5e83b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" podUID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" Nov 03 10:30:16.663516 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:16.663459 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2"] Nov 03 10:30:16.814328 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:16.814302 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" Nov 03 10:30:16.904924 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:30:16.904861 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:30:16.954915 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:16.954835 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70de685-f6c7-4ae8-8b67-623d11b5e83b-kube-api-access-bw6mz\" (UniqueName: \"kubernetes.io/projected/a70de685-f6c7-4ae8-8b67-623d11b5e83b-kube-api-access-bw6mz\") pod \"a70de685-f6c7-4ae8-8b67-623d11b5e83b\" (UID: \"a70de685-f6c7-4ae8-8b67-623d11b5e83b\") " Nov 03 10:30:16.955093 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:16.954919 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70de685-f6c7-4ae8-8b67-623d11b5e83b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70de685-f6c7-4ae8-8b67-623d11b5e83b-maas-api-service-ca\") pod \"a70de685-f6c7-4ae8-8b67-623d11b5e83b\" (UID: \"a70de685-f6c7-4ae8-8b67-623d11b5e83b\") " Nov 03 10:30:16.955538 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:16.955507 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70de685-f6c7-4ae8-8b67-623d11b5e83b-maas-api-service-ca" pod "a70de685-f6c7-4ae8-8b67-623d11b5e83b" (UID: "a70de685-f6c7-4ae8-8b67-623d11b5e83b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:30:16.957869 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:16.957845 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70de685-f6c7-4ae8-8b67-623d11b5e83b-kube-api-access-bw6mz" pod "a70de685-f6c7-4ae8-8b67-623d11b5e83b" (UID: "a70de685-f6c7-4ae8-8b67-623d11b5e83b"). InnerVolumeSpecName "kube-api-access-bw6mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:30:17.055863 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:17.055826 5015 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70de685-f6c7-4ae8-8b67-623d11b5e83b-maas-api-service-ca\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:30:17.055863 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:17.055860 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw6mz\" (UniqueName: \"kubernetes.io/projected/a70de685-f6c7-4ae8-8b67-623d11b5e83b-kube-api-access-bw6mz\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:30:17.554161 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:17.554121 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" event={"ID":"a70de685-f6c7-4ae8-8b67-623d11b5e83b","Type":"ContainerDied","Data":"72ea1eb0b6016713905cac4cab6e16367259ec1948a7fa5573377de8b4e5c682"} Nov 03 10:30:17.554362 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:17.554190 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2" Nov 03 10:30:17.554362 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:17.554194 5015 scope.go:122] "RemoveContainer" containerID="e4eed314cd2da1ff88969af74b5247c13f9d3a310c79ea911481fef7778fdaf1" Nov 03 10:30:17.726504 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:17.726450 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2"] Nov 03 10:30:17.776431 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:17.776398 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895030-gzjs2"] Nov 03 10:30:18.562550 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:18.562510 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5d4t6/must-gather-tm2kj" event={"ID":"61d5bca4-1b98-4866-b0c0-f0bad41a5e7f","Type":"ContainerStarted","Data":"9159ee69849e2992ad56eea18611e79ceae506f1e47e5d9fc7bf88ac0638cbc6"} Nov 03 10:30:18.562756 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:18.562562 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5d4t6/must-gather-tm2kj" event={"ID":"61d5bca4-1b98-4866-b0c0-f0bad41a5e7f","Type":"ContainerStarted","Data":"440f49e7008839fa15a416ebfc84a33c0e4fb75eb4329dd8af0e8a211af19dcd"} Nov 03 10:30:18.664653 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:18.664623 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c9fnb_25b42294-fcbd-4bc8-8652-9d4d7e762e3d/global-pull-secret-syncer/0.log" Nov 03 10:30:18.679728 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:18.679679 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5d4t6/must-gather-tm2kj" podStartSLOduration=2.01527631 podStartE2EDuration="9.679664953s" totalImagesPullingTime="7.664388643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:30:18.677825354 +0000 UTC m=+6651.602767572" watchObservedRunningTime="2026-11-03 10:30:18.679664953 +0000 UTC m=+6651.604607168" Nov 03 10:30:19.203801 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:19.203772 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" path="/var/lib/kubelet/pods/a70de685-f6c7-4ae8-8b67-623d11b5e83b/volumes" Nov 03 10:30:20.456675 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:20.456646 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-27vdv_248b4530-5359-4e26-8309-78d1a7128494/connection-checker/0.log" Nov 03 10:30:20.881007 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:20.880982 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5qm4n_61655665-1941-4f5f-bc6b-41e51b2b3774/connection-checker/0.log" Nov 03 10:30:21.305964 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:21.305938 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-p5ncs_c41e7f71-1eaa-4208-9fd8-dc6a2bee1e0a/connection-checker/0.log" Nov 03 10:30:22.959500 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:22.959435 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hp76w_1330da31-fcb6-4c7b-9db7-0d4d96262234/konnectivity-agent/0.log" Nov 03 10:30:22.984133 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:22.984101 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wwbxd_e6a48fda-d04e-423f-89dc-a640c4b2d513/manager/0.log" Nov 03 10:30:23.975672 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:23.975639 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-14.us-west-1.compute.internal_6652f123a319d675474cd7afd732c542/haproxy/0.log" Nov 03 10:30:24.054070 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:24.054032 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm"] Nov 03 10:30:24.103881 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:24.103847 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189eb33c-hj2zm"] Nov 03 10:30:25.203502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:25.203435 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355aac11-afb8-4865-a7f1-50db3ec3d7d2" path="/var/lib/kubelet/pods/355aac11-afb8-4865-a7f1-50db3ec3d7d2/volumes" Nov 03 10:30:29.014280 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:29.014239 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wwbxd_e6a48fda-d04e-423f-89dc-a640c4b2d513/manager/0.log" Nov 03 10:30:30.404524 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:30.404493 5015 scope.go:122] "RemoveContainer" containerID="cc5ab99358d2286ff7ca776c6facc540f9c33ccbd3509fb7b788ad99bb6b7d4c" Nov 03 10:30:30.414697 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:30.414678 5015 scope.go:122] "RemoveContainer" containerID="da7800a9ade9d0fea0fda4f5d51f9d03373582be8108772da51acda61c303265" Nov 03 10:30:35.054275 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:35.054235 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26"] Nov 03 10:30:35.098901 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:35.098867 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wwbxd_e6a48fda-d04e-423f-89dc-a640c4b2d513/manager/0.log" Nov 03 10:30:35.103286 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:35.103259 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a4d67c-bcn26"] Nov 03 10:30:35.203189 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:35.203146 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85404c28-3ba7-45f8-a61b-6e726be726e9" path="/var/lib/kubelet/pods/85404c28-3ba7-45f8-a61b-6e726be726e9/volumes" Nov 03 10:30:40.934773 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:40.934741 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-86rlh_af4db30f-c036-4273-b29e-733cd2dbe65f/istio-proxy/0.log" Nov 03 10:30:43.855312 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:43.855267 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-dhjmc_ded8f558-e327-4178-a2ac-948e7f005ef1/istio-proxy/0.log" Nov 03 10:30:44.729834 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:44.729801 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lg5gm_0d1bcdca-f1fe-45db-98df-66f3ebd5696e/istio-proxy/0.log" Nov 03 10:30:44.948958 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:44.948921 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lkg9f_fdb2f094-67e8-438b-80b8-c1ef4521b355/istio-proxy/0.log" Nov 03 10:30:45.160995 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:45.160908 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7_f9d5e9a1-bee4-4750-90bf-892c58ff29a5/istio-proxy/0.log" Nov 03 10:30:45.375090 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:45.375052 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cf9dd8f69-nc6dw_6d2b9c62-2c04-43b1-a529-cfb16c231ecd/payload-pre-processing/0.log" Nov 03 10:30:45.591741 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:45.591711 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85f55897bb-jwz7j_8b000dd7-72c1-491d-8500-6acde8c672b0/payload-processing/0.log" Nov 03 10:30:45.818342 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:45.818312 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b64c9d49-xhzz8_70bfc17c-8e18-473a-a8fe-e386112cfaa4/router/0.log" Nov 03 10:30:46.840524 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:46.840462 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dfcc7c588-7hvqz_ae051d13-de71-4752-b918-8bb60b21002d/authorino/0.log" Nov 03 10:30:46.902133 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:30:46.902093 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:30:48.043596 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:48.043562 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rc42m_1786e9e7-2202-4fce-aaf4-2f0b30932688/manager/0.log" Nov 03 10:30:49.388786 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:49.388745 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cr2kg_19f7d098-9626-4724-a15e-85253b3dfd54/manager/0.log" Nov 03 10:30:50.924997 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:50.924941 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dfcc7c588-7hvqz_ae051d13-de71-4752-b918-8bb60b21002d/authorino/0.log" Nov 03 10:30:51.041320 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:51.041279 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rc42m_1786e9e7-2202-4fce-aaf4-2f0b30932688/manager/0.log" Nov 03 10:30:51.751753 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:51.751700 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cr2kg_19f7d098-9626-4724-a15e-85253b3dfd54/manager/0.log" Nov 03 10:30:56.131910 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:56.131868 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-jrqm5_57491634-c53c-4571-854c-ece83d3f09a2/cluster-monitoring-operator/0.log" Nov 03 10:30:57.015028 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.013456 5015 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c"] Nov 03 10:30:57.015028 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.014541 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" containerName="cleanup" Nov 03 10:30:57.015028 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.014563 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" containerName="cleanup" Nov 03 10:30:57.015028 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.014579 5015 cpu_manager.go:414] "containerMap: removing container" podUID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" containerName="cleanup" Nov 03 10:30:57.015028 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.014588 5015 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" containerName="cleanup" Nov 03 10:30:57.015620 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.015599 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" containerName="cleanup" Nov 03 10:30:57.015728 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.015717 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" containerName="cleanup" Nov 03 10:30:57.015843 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.015831 5015 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70de685-f6c7-4ae8-8b67-623d11b5e83b" containerName="cleanup" Nov 03 10:30:57.021729 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.021696 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.075813 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.075774 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b4k9m\"/\"kube-root-ca.crt\"" Nov 03 10:30:57.126368 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.126333 5015 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b4k9m\"/\"default-dockercfg-xt4bq\"" Nov 03 10:30:57.126368 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.126357 5015 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b4k9m\"/\"openshift-service-ca.crt\"" Nov 03 10:30:57.130209 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.130178 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c"] Nov 03 10:30:57.172547 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.172505 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-lib-modules\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.173017 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.172735 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8llcn\" (UniqueName: \"kubernetes.io/projected/e77b38c1-eafb-4673-85cf-7fb5d7be7809-kube-api-access-8llcn\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.173017 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.172813 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-sys\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.173017 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.172859 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-podres\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.173017 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.172943 5015 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-proc\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.273646 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.273547 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-lib-modules\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.273646 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.273591 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8llcn\" (UniqueName: \"kubernetes.io/projected/e77b38c1-eafb-4673-85cf-7fb5d7be7809-kube-api-access-8llcn\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.273646 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.273610 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-sys\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.273961 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.273701 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-sys\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.273961 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.273714 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-lib-modules\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.273961 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.273760 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-podres\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.273961 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.273834 5015 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-proc\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.273961 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.273863 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-podres\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.273961 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.273926 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e77b38c1-eafb-4673-85cf-7fb5d7be7809-proc\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.333030 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.332990 5015 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8llcn\" (UniqueName: \"kubernetes.io/projected/e77b38c1-eafb-4673-85cf-7fb5d7be7809-kube-api-access-8llcn\") pod \"perf-node-gather-daemonset-fc97c\" (UID: \"e77b38c1-eafb-4673-85cf-7fb5d7be7809\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.336938 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.336918 5015 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:57.713274 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.713240 5015 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c"] Nov 03 10:30:57.714151 ip-10-0-131-14 kubenswrapper[5015]: W1103 10:30:57.714116 5015 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode77b38c1_eafb_4673_85cf_7fb5d7be7809.slice/crio-403f7f5b2cad1fe28fb325f7e1cc932f3a8f25e9d807e510e96f5f0769339569 WatchSource:0}: Error finding container 403f7f5b2cad1fe28fb325f7e1cc932f3a8f25e9d807e510e96f5f0769339569: Status 404 returned error can't find the container with id 403f7f5b2cad1fe28fb325f7e1cc932f3a8f25e9d807e510e96f5f0769339569 Nov 03 10:30:57.833995 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:57.833954 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" event={"ID":"e77b38c1-eafb-4673-85cf-7fb5d7be7809","Type":"ContainerStarted","Data":"403f7f5b2cad1fe28fb325f7e1cc932f3a8f25e9d807e510e96f5f0769339569"} Nov 03 10:30:58.729522 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:58.729462 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frljt_dafdc498-1da7-4acb-9a07-a95413ed93d0/node-exporter/0.log" Nov 03 10:30:58.841730 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:58.841692 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" event={"ID":"e77b38c1-eafb-4673-85cf-7fb5d7be7809","Type":"ContainerStarted","Data":"98d33c131d54aa9a46e4bcc91a94224412253a082546e02f732d92f278b4e3d7"} Nov 03 10:30:58.841929 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:58.841791 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:30:58.959757 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:58.959702 5015 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" podStartSLOduration=2.959687679 podStartE2EDuration="2.959687679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:30:58.95625286 +0000 UTC m=+6691.881195077" watchObservedRunningTime="2026-11-03 10:30:58.959687679 +0000 UTC m=+6691.884629895" Nov 03 10:30:59.150925 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:59.150821 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frljt_dafdc498-1da7-4acb-9a07-a95413ed93d0/kube-rbac-proxy/0.log" Nov 03 10:30:59.575937 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:30:59.575909 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frljt_dafdc498-1da7-4acb-9a07-a95413ed93d0/init-textfile/0.log" Nov 03 10:31:04.860051 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:04.860022 5015 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-fc97c" Nov 03 10:31:14.424219 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:14.424184 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-vrpjd_283e4bdb-5662-4b18-8775-7c86dd4323f3/networking-console-plugin/0.log" Nov 03 10:31:16.902620 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:31:16.902581 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:31:17.168640 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:17.168557 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:31:17.176068 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:17.176034 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/4.log" Nov 03 10:31:21.018708 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:21.018679 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-dplnt_0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2/manager/0.log" Nov 03 10:31:21.231027 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:21.230993 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-dplnt_0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2/copy-manifests/0.log" Nov 03 10:31:22.849673 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:22.849645 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-b7vq2_31ed67dd-1988-47af-a5a6-e00ee72236a3/volume-data-source-validator/0.log" Nov 03 10:31:24.624176 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:24.624139 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wwbxd_e6a48fda-d04e-423f-89dc-a640c4b2d513/manager/0.log" Nov 03 10:31:25.051935 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:25.051897 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r"] Nov 03 10:31:25.101880 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:25.101844 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861756ce-tr49r"] Nov 03 10:31:25.203118 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:25.203083 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487e8c78-f695-4dcf-aa4e-b8668a69c646" path="/var/lib/kubelet/pods/487e8c78-f695-4dcf-aa4e-b8668a69c646/volumes" Nov 03 10:31:29.162279 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:29.162250 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6rpvp_587a6bc5-67cb-469b-b4b9-b35da4fb382b/dns/0.log" Nov 03 10:31:29.412902 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:29.412818 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-86rlh_af4db30f-c036-4273-b29e-733cd2dbe65f/istio-proxy/0.log" Nov 03 10:31:29.581007 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:29.580979 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6rpvp_587a6bc5-67cb-469b-b4b9-b35da4fb382b/kube-rbac-proxy/0.log" Nov 03 10:31:30.451667 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:30.451635 5015 scope.go:122] "RemoveContainer" containerID="5cdb0536c3aac117d2749fc25cd28a441bacf24c80db8b93f447322a308ea3d7" Nov 03 10:31:30.462858 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:30.462834 5015 scope.go:122] "RemoveContainer" containerID="924587d3163b61b7d4be4c4065d15dbb5416be5acc1835ad55466eb5e8275841" Nov 03 10:31:30.860093 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:30.860059 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2xrz9_70d5c7ae-f8aa-4300-b5a6-c76273b66b3c/dns-node-resolver/0.log" Nov 03 10:31:32.040288 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:32.040252 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-dhjmc_ded8f558-e327-4178-a2ac-948e7f005ef1/istio-proxy/0.log" Nov 03 10:31:32.910658 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:32.910622 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lg5gm_0d1bcdca-f1fe-45db-98df-66f3ebd5696e/istio-proxy/0.log" Nov 03 10:31:33.129387 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:33.129358 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lkg9f_fdb2f094-67e8-438b-80b8-c1ef4521b355/istio-proxy/0.log" Nov 03 10:31:33.341700 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:33.341671 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7_f9d5e9a1-bee4-4750-90bf-892c58ff29a5/istio-proxy/0.log" Nov 03 10:31:33.557023 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:33.556989 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cf9dd8f69-nc6dw_6d2b9c62-2c04-43b1-a529-cfb16c231ecd/payload-pre-processing/0.log" Nov 03 10:31:33.771328 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:33.771297 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85f55897bb-jwz7j_8b000dd7-72c1-491d-8500-6acde8c672b0/payload-processing/0.log" Nov 03 10:31:33.995108 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:33.995075 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b64c9d49-xhzz8_70bfc17c-8e18-473a-a8fe-e386112cfaa4/router/0.log" Nov 03 10:31:35.971161 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:35.971135 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wkb5w_d3a0d3db-7b81-4e10-b240-b665c083bda2/node-ca/0.log" Nov 03 10:31:36.053331 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:36.053292 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml"] Nov 03 10:31:36.103314 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:36.103279 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0fd88da-c4bml"] Nov 03 10:31:37.203279 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:37.203245 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72de1663-8c9b-4c9b-b186-8e2872f1622b" path="/var/lib/kubelet/pods/72de1663-8c9b-4c9b-b186-8e2872f1622b/volumes" Nov 03 10:31:37.460265 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:37.460188 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-74879684bf-qbl6q_250a2b15-6da6-45dc-86fa-55bbe5752688/manager/0.log" Nov 03 10:31:40.161228 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:40.161190 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-86rlh_af4db30f-c036-4273-b29e-733cd2dbe65f/istio-proxy/0.log" Nov 03 10:31:40.499837 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:40.499810 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-86rlh_af4db30f-c036-4273-b29e-733cd2dbe65f/istio-proxy/0.log" Nov 03 10:31:43.320888 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:43.320856 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-dhjmc_ded8f558-e327-4178-a2ac-948e7f005ef1/istio-proxy/0.log" Nov 03 10:31:44.188435 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:44.188396 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lg5gm_0d1bcdca-f1fe-45db-98df-66f3ebd5696e/istio-proxy/0.log" Nov 03 10:31:44.406956 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:44.406913 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lkg9f_fdb2f094-67e8-438b-80b8-c1ef4521b355/istio-proxy/0.log" Nov 03 10:31:44.621697 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:44.621667 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7_f9d5e9a1-bee4-4750-90bf-892c58ff29a5/istio-proxy/0.log" Nov 03 10:31:44.836459 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:44.836427 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cf9dd8f69-nc6dw_6d2b9c62-2c04-43b1-a529-cfb16c231ecd/payload-pre-processing/0.log" Nov 03 10:31:45.051984 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:45.051951 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85f55897bb-jwz7j_8b000dd7-72c1-491d-8500-6acde8c672b0/payload-processing/0.log" Nov 03 10:31:45.278661 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:45.278626 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b64c9d49-xhzz8_70bfc17c-8e18-473a-a8fe-e386112cfaa4/router/0.log" Nov 03 10:31:46.155500 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:46.154848 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-dhjmc_ded8f558-e327-4178-a2ac-948e7f005ef1/istio-proxy/0.log" Nov 03 10:31:46.913040 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:31:46.913001 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:31:47.054666 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:47.054634 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh"] Nov 03 10:31:47.105943 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:47.105907 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9aafc4-xp7hh"] Nov 03 10:31:47.207663 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:47.207585 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc612f6-eb87-424c-ad70-07b5581e27a4" path="/var/lib/kubelet/pods/cdc612f6-eb87-424c-ad70-07b5581e27a4/volumes" Nov 03 10:31:47.876671 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:47.876643 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lg5gm_0d1bcdca-f1fe-45db-98df-66f3ebd5696e/istio-proxy/0.log" Nov 03 10:31:48.308125 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:48.308091 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lkg9f_fdb2f094-67e8-438b-80b8-c1ef4521b355/istio-proxy/0.log" Nov 03 10:31:48.729813 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:48.729784 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7_f9d5e9a1-bee4-4750-90bf-892c58ff29a5/istio-proxy/0.log" Nov 03 10:31:49.155006 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:49.154919 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cf9dd8f69-nc6dw_6d2b9c62-2c04-43b1-a529-cfb16c231ecd/payload-pre-processing/0.log" Nov 03 10:31:49.580775 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:49.580744 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85f55897bb-jwz7j_8b000dd7-72c1-491d-8500-6acde8c672b0/payload-processing/0.log" Nov 03 10:31:50.014681 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:50.014648 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b64c9d49-xhzz8_70bfc17c-8e18-473a-a8fe-e386112cfaa4/router/0.log" Nov 03 10:31:50.191244 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:50.191212 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-86rlh_af4db30f-c036-4273-b29e-733cd2dbe65f/istio-proxy/0.log" Nov 03 10:31:52.928696 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:52.928649 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-dhjmc_ded8f558-e327-4178-a2ac-948e7f005ef1/istio-proxy/0.log" Nov 03 10:31:53.805304 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:53.805264 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lg5gm_0d1bcdca-f1fe-45db-98df-66f3ebd5696e/istio-proxy/0.log" Nov 03 10:31:54.024498 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:54.024059 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-lkg9f_fdb2f094-67e8-438b-80b8-c1ef4521b355/istio-proxy/0.log" Nov 03 10:31:54.237936 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:54.237897 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-pl6j7_f9d5e9a1-bee4-4750-90bf-892c58ff29a5/istio-proxy/0.log" Nov 03 10:31:54.451460 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:54.451429 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6cf9dd8f69-nc6dw_6d2b9c62-2c04-43b1-a529-cfb16c231ecd/payload-pre-processing/0.log" Nov 03 10:31:54.667322 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:54.667230 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85f55897bb-jwz7j_8b000dd7-72c1-491d-8500-6acde8c672b0/payload-processing/0.log" Nov 03 10:31:54.860466 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:54.859113 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xs65g_1bedc7c0-f94a-446e-8029-c3e82609f30f/serve-healthcheck-canary/0.log" Nov 03 10:31:54.894873 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:54.894844 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b64c9d49-xhzz8_70bfc17c-8e18-473a-a8fe-e386112cfaa4/router/0.log" Nov 03 10:31:57.381621 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:57.381588 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-b4ggb_22edddd5-3b05-4942-910a-651d537867c8/insights-operator/0.log" Nov 03 10:31:57.385187 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:57.385161 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-b4ggb_22edddd5-3b05-4942-910a-651d537867c8/insights-operator/1.log" Nov 03 10:31:57.751107 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:57.751076 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pc6m2_19c1fbeb-055b-4c85-aebe-d4856252042a/kube-rbac-proxy/0.log" Nov 03 10:31:58.178575 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:58.177287 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pc6m2_19c1fbeb-055b-4c85-aebe-d4856252042a/exporter/0.log" Nov 03 10:31:58.602918 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:58.601627 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pc6m2_19c1fbeb-055b-4c85-aebe-d4856252042a/extractor/0.log" Nov 03 10:31:59.056296 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:59.056237 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt"] Nov 03 10:31:59.103946 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:59.103913 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3028a06a-299zt"] Nov 03 10:31:59.204860 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:31:59.204825 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87493051-b17d-4aaa-8674-dbf6575b059d" path="/var/lib/kubelet/pods/87493051-b17d-4aaa-8674-dbf6575b059d/volumes" Nov 03 10:32:01.586500 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:01.585813 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-r6xbr-hw6qv_d5cbd134-00f7-43d1-9890-960de193c02f/insights-gathering/0.log" Nov 03 10:32:07.212974 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:07.212937 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-dplnt_0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2/manager/0.log" Nov 03 10:32:07.638165 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:07.638126 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-dplnt_0e602fbb-7c1e-41ef-8c9e-a174df7dc1d2/copy-manifests/0.log" Nov 03 10:32:10.053355 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:10.053304 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq"] Nov 03 10:32:10.104512 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:10.104444 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11318af9-rnqdq"] Nov 03 10:32:11.204489 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:11.204446 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="249c8c10-8e03-41c6-997a-c413fff553e0" path="/var/lib/kubelet/pods/249c8c10-8e03-41c6-997a-c413fff553e0/volumes" Nov 03 10:32:11.650976 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:11.650944 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-74879684bf-qbl6q_250a2b15-6da6-45dc-86fa-55bbe5752688/manager/0.log" Nov 03 10:32:14.948915 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:14.948879 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dfcc7c588-7hvqz_ae051d13-de71-4752-b918-8bb60b21002d/authorino/0.log" Nov 03 10:32:15.079894 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:15.079863 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rc42m_1786e9e7-2202-4fce-aaf4-2f0b30932688/manager/0.log" Nov 03 10:32:15.693553 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:15.693518 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wwbxd_e6a48fda-d04e-423f-89dc-a640c4b2d513/manager/0.log" Nov 03 10:32:15.792762 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:15.792730 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cr2kg_19f7d098-9626-4724-a15e-85253b3dfd54/manager/0.log" Nov 03 10:32:16.901413 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:32:16.901374 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:32:20.067136 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:20.067098 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks"] Nov 03 10:32:20.119508 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:20.119455 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5557744-rr2ks"] Nov 03 10:32:21.204465 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:21.204434 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a" path="/var/lib/kubelet/pods/8b1c33e6-6e6c-4676-b84e-4a2226d1bb4a/volumes" Nov 03 10:32:30.501106 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:30.501071 5015 scope.go:122] "RemoveContainer" containerID="d009005cc69a6dced323a556d3dcd5caba6714433189994def540f51cc3795c6" Nov 03 10:32:30.511757 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:30.511722 5015 scope.go:122] "RemoveContainer" containerID="7e0a63bb050e4efa9e8f5b267df4dbc6a51859bcaadf828dfe4cb9463e007161" Nov 03 10:32:31.094458 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:31.094418 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc"] Nov 03 10:32:31.097162 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:31.097133 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b0b816-sfmjc"] Nov 03 10:32:31.208518 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:31.208455 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10eabecf-4c83-4930-b92a-e478440f283d" path="/var/lib/kubelet/pods/10eabecf-4c83-4930-b92a-e478440f283d/volumes" Nov 03 10:32:38.909667 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:38.909575 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ghcc7_3a650ee8-e4e1-4c94-a7da-836ea74de483/migrator/0.log" Nov 03 10:32:39.335162 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:39.335121 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ldh5h_bfcdc2e6-1f23-4da9-a478-0ad2175c5def/migrator/0.log" Nov 03 10:32:41.682910 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:41.682874 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-ck58r_df61dbd1-26b9-4d1b-847c-785687d935ab/kube-storage-version-migrator-operator/1.log" Nov 03 10:32:41.684396 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:41.684369 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-ck58r_df61dbd1-26b9-4d1b-847c-785687d935ab/kube-storage-version-migrator-operator/0.log" Nov 03 10:32:42.052038 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:42.051996 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h"] Nov 03 10:32:42.102174 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:42.102131 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fa3f8f-scl9h"] Nov 03 10:32:43.204129 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:43.204093 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41" path="/var/lib/kubelet/pods/25ed9e13-f95d-45e0-8ec9-0a6d5a6e2b41/volumes" Nov 03 10:32:46.911596 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:32:46.911555 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:32:47.577936 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:47.577902 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8hwgr_39df63cd-3b12-4930-8a23-04ed67d961a8/kube-multus/0.log" Nov 03 10:32:48.126234 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:48.126203 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4chsq_0893a4fe-c6b7-42e5-8e5c-8d179dfefedc/kube-multus-additional-cni-plugins/0.log" Nov 03 10:32:48.551021 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:48.550976 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4chsq_0893a4fe-c6b7-42e5-8e5c-8d179dfefedc/egress-router-binary-copy/0.log" Nov 03 10:32:48.978017 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:48.976319 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4chsq_0893a4fe-c6b7-42e5-8e5c-8d179dfefedc/cni-plugins/0.log" Nov 03 10:32:49.400718 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:49.400626 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4chsq_0893a4fe-c6b7-42e5-8e5c-8d179dfefedc/bond-cni-plugin/0.log" Nov 03 10:32:49.823093 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:49.823066 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4chsq_0893a4fe-c6b7-42e5-8e5c-8d179dfefedc/routeoverride-cni/0.log" Nov 03 10:32:50.246091 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:50.246064 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4chsq_0893a4fe-c6b7-42e5-8e5c-8d179dfefedc/whereabouts-cni-bincopy/0.log" Nov 03 10:32:50.670684 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:50.670579 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4chsq_0893a4fe-c6b7-42e5-8e5c-8d179dfefedc/whereabouts-cni/0.log" Nov 03 10:32:50.838666 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:50.838630 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dfcc7c588-7hvqz_ae051d13-de71-4752-b918-8bb60b21002d/authorino/0.log" Nov 03 10:32:50.973395 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:50.973365 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rc42m_1786e9e7-2202-4fce-aaf4-2f0b30932688/manager/0.log" Nov 03 10:32:51.683097 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:51.683063 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cr2kg_19f7d098-9626-4724-a15e-85253b3dfd54/manager/0.log" Nov 03 10:32:52.052300 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:52.052258 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp"] Nov 03 10:32:52.102419 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:52.102385 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60760ce0-h7shp"] Nov 03 10:32:53.205909 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:53.205876 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b816de8-c9e0-4130-863d-6463e1e3c71a" path="/var/lib/kubelet/pods/5b816de8-c9e0-4130-863d-6463e1e3c71a/volumes" Nov 03 10:32:59.519555 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:32:59.519520 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q7vbp_63810bfe-e55b-4524-bde4-8f072b4b63a3/network-metrics-daemon/0.log" Nov 03 10:33:00.014513 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:00.014443 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q7vbp_63810bfe-e55b-4524-bde4-8f072b4b63a3/kube-rbac-proxy/0.log" Nov 03 10:33:03.055259 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:03.055201 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln"] Nov 03 10:33:03.104346 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:03.104313 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686b0d6f-dstln"] Nov 03 10:33:03.203882 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:03.203851 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702136bd-30d2-41bd-a7f4-d6fdcfc47d66" path="/var/lib/kubelet/pods/702136bd-30d2-41bd-a7f4-d6fdcfc47d66/volumes" Nov 03 10:33:03.904023 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:03.903996 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-qrhwd_6f925e33-6569-4ac4-9422-1ac38473de7a/main/0.log" Nov 03 10:33:04.337314 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:04.337271 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8_12c4a382-3942-493e-84a4-85d061fc87e4/main/0.log" Nov 03 10:33:04.807075 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:04.807045 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9884s_4537f2eb-9880-4ba9-a64a-0c25e21521dd/main/0.log" Nov 03 10:33:05.024113 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:05.024083 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6_21ad0cfe-e3c6-4770-b0b3-7c512e7f1830/main/0.log" Nov 03 10:33:08.006502 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:08.005968 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fnh7l_1959beea-2daf-4e4a-a412-9bc2d35fc277/ovn-controller/0.log" Nov 03 10:33:08.528763 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:08.528719 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fnh7l_1959beea-2daf-4e4a-a412-9bc2d35fc277/ovn-acl-logging/0.log" Nov 03 10:33:08.954336 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:08.954237 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fnh7l_1959beea-2daf-4e4a-a412-9bc2d35fc277/kube-rbac-proxy-node/0.log" Nov 03 10:33:09.377614 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:09.377581 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fnh7l_1959beea-2daf-4e4a-a412-9bc2d35fc277/kube-rbac-proxy-ovn-metrics/0.log" Nov 03 10:33:09.797760 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:09.797727 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fnh7l_1959beea-2daf-4e4a-a412-9bc2d35fc277/northd/0.log" Nov 03 10:33:10.222217 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:10.222192 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fnh7l_1959beea-2daf-4e4a-a412-9bc2d35fc277/nbdb/0.log" Nov 03 10:33:10.650418 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:10.650333 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fnh7l_1959beea-2daf-4e4a-a412-9bc2d35fc277/sbdb/0.log" Nov 03 10:33:11.324023 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:11.323988 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fnh7l_1959beea-2daf-4e4a-a412-9bc2d35fc277/ovnkube-controller/0.log" Nov 03 10:33:14.053391 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:14.053352 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t"] Nov 03 10:33:14.059278 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:14.059253 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53f53c1-tlp9t"] Nov 03 10:33:15.204764 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:15.204718 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6355a158-af38-4f4c-8ae7-afa79ee6ce1b" path="/var/lib/kubelet/pods/6355a158-af38-4f4c-8ae7-afa79ee6ce1b/volumes" Nov 03 10:33:16.910069 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:33:16.910026 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:33:20.965764 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:20.965733 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f78d59f99-kvwlg_204918bd-5b87-4358-b80c-cd6262afb3b0/check-endpoints/0.log" Nov 03 10:33:21.389291 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:21.389260 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-4cbq5_f70226b0-5f4c-465c-9ec7-7ceee1bb376f/network-check-target-container/0.log" Nov 03 10:33:24.095808 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:24.095775 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd"] Nov 03 10:33:24.102428 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:24.102388 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00a990b0-ssctd"] Nov 03 10:33:25.203882 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:25.203843 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4b3383-4c1c-4de5-94ea-43aa01896169" path="/var/lib/kubelet/pods/5f4b3383-4c1c-4de5-94ea-43aa01896169/volumes" Nov 03 10:33:27.015820 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:27.015772 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dfcc7c588-7hvqz_ae051d13-de71-4752-b918-8bb60b21002d/authorino/0.log" Nov 03 10:33:27.146788 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:27.146760 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rc42m_1786e9e7-2202-4fce-aaf4-2f0b30932688/manager/0.log" Nov 03 10:33:27.859443 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:27.859349 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cr2kg_19f7d098-9626-4724-a15e-85253b3dfd54/manager/0.log" Nov 03 10:33:29.734501 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:29.734443 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zk7xq_07f9e882-e987-44b0-a688-45e3c9abec5c/iptables-alerter/0.log" Nov 03 10:33:30.548443 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:30.548413 5015 scope.go:122] "RemoveContainer" containerID="dad1a27d960bcddc8a0a572c61bb92914da02d25939b410211c2d730962e120f" Nov 03 10:33:30.562162 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:30.562142 5015 scope.go:122] "RemoveContainer" containerID="b36faa9d1cd36c1381f3a44cf39477c124d191e4bb9e564bf4d3c3270c9dc91f" Nov 03 10:33:30.572387 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:30.572363 5015 scope.go:122] "RemoveContainer" containerID="cee01cf0730c04716a9a210b837eb33878e6f2b75b723404881fbe43e885f334" Nov 03 10:33:30.584221 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:30.584148 5015 scope.go:122] "RemoveContainer" containerID="280d073c58446dd517e3e297c8b682024b00628a75256820404b2441ed5a88d3" Nov 03 10:33:30.593436 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:30.593403 5015 scope.go:122] "RemoveContainer" containerID="c10420dbb4d1dab4835959f5503a2aa042087cd6cf75d604362b0d97811db7cd" Nov 03 10:33:30.602751 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:30.602721 5015 scope.go:122] "RemoveContainer" containerID="436ecd695b07986d48ff6c4e156a24cc1f62201a812d78f8f9dfa2e79f13a222" Nov 03 10:33:34.081073 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:34.081038 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9qzgm_83c6ab8c-6c62-4163-a789-1edb75af2ad9/tuned/0.log" Nov 03 10:33:41.800128 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:41.800006 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-qrhwd_6f925e33-6569-4ac4-9422-1ac38473de7a/main/0.log" Nov 03 10:33:42.234309 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:42.234277 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8_12c4a382-3942-493e-84a4-85d061fc87e4/main/0.log" Nov 03 10:33:42.667160 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:42.667083 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9884s_4537f2eb-9880-4ba9-a64a-0c25e21521dd/main/0.log" Nov 03 10:33:42.884003 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:42.883973 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6_21ad0cfe-e3c6-4770-b0b3-7c512e7f1830/main/0.log" Nov 03 10:33:46.232951 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:46.232912 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-jgcwt_7bebbc2e-c649-4f7f-bef3-0a8f158d6cae/cluster-samples-operator/0.log" Nov 03 10:33:46.732309 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:46.732277 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-jgcwt_7bebbc2e-c649-4f7f-bef3-0a8f158d6cae/cluster-samples-operator-watch/0.log" Nov 03 10:33:46.906285 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:33:46.906248 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:33:51.513347 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:51.513311 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-544d749c8c-s5kxx_16cb59f2-9d9f-4904-9fc8-bed9111deecd/service-ca-operator/1.log" Nov 03 10:33:51.519286 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:51.519258 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-544d749c8c-s5kxx_16cb59f2-9d9f-4904-9fc8-bed9111deecd/service-ca-operator/0.log" Nov 03 10:33:56.570556 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:56.570521 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-82g5z_bfbee40d-3926-4eae-babe-e598bfae8098/csi-driver/0.log" Nov 03 10:33:56.992678 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:56.992639 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-82g5z_bfbee40d-3926-4eae-babe-e598bfae8098/csi-node-driver-registrar/0.log" Nov 03 10:33:57.417878 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:33:57.417796 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-82g5z_bfbee40d-3926-4eae-babe-e598bfae8098/csi-liveness-probe/0.log" Nov 03 10:34:03.393653 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:03.393609 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dfcc7c588-7hvqz_ae051d13-de71-4752-b918-8bb60b21002d/authorino/0.log" Nov 03 10:34:03.519740 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:03.519688 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rc42m_1786e9e7-2202-4fce-aaf4-2f0b30932688/manager/0.log" Nov 03 10:34:04.231621 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:04.231583 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cr2kg_19f7d098-9626-4724-a15e-85253b3dfd54/manager/0.log" Nov 03 10:34:16.905246 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:34:16.905159 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:34:21.998543 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:21.998507 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc"] Nov 03 10:34:21.999189 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:21.998776 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerName="istio-proxy" containerID="cri-o://edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690" gracePeriod=30 Nov 03 10:34:22.047013 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:22.046973 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm"] Nov 03 10:34:22.047215 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:22.047170 5015 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerName="istio-proxy" containerID="cri-o://0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd" gracePeriod=30 Nov 03 10:34:27.310942 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.310917 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:34:27.351547 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.351522 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:34:27.360204 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360178 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-podinfo\") pod \"ded8f558-e327-4178-a2ac-948e7f005ef1\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " Nov 03 10:34:27.360339 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360222 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-certs\") pod \"ded8f558-e327-4178-a2ac-948e7f005ef1\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " Nov 03 10:34:27.360339 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360264 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded8f558-e327-4178-a2ac-948e7f005ef1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded8f558-e327-4178-a2ac-948e7f005ef1-istiod-ca-cert\") pod \"ded8f558-e327-4178-a2ac-948e7f005ef1\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " Nov 03 10:34:27.360339 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360295 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-envoy\") pod \"ded8f558-e327-4178-a2ac-948e7f005ef1\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " Nov 03 10:34:27.360339 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360334 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-data\") pod \"ded8f558-e327-4178-a2ac-948e7f005ef1\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " Nov 03 10:34:27.360592 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360388 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-credential-socket\") pod \"ded8f558-e327-4178-a2ac-948e7f005ef1\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " Nov 03 10:34:27.360592 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360422 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-kube-api-access-hf5m8\" (UniqueName: \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-kube-api-access-hf5m8\") pod \"ded8f558-e327-4178-a2ac-948e7f005ef1\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " Nov 03 10:34:27.360592 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360504 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-token\" (UniqueName: \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-token\") pod \"ded8f558-e327-4178-a2ac-948e7f005ef1\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " Nov 03 10:34:27.360592 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360553 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-socket\") pod \"ded8f558-e327-4178-a2ac-948e7f005ef1\" (UID: \"ded8f558-e327-4178-a2ac-948e7f005ef1\") " Nov 03 10:34:27.360831 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360676 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-certs" pod "ded8f558-e327-4178-a2ac-948e7f005ef1" (UID: "ded8f558-e327-4178-a2ac-948e7f005ef1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.360831 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360698 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded8f558-e327-4178-a2ac-948e7f005ef1-istiod-ca-cert" pod "ded8f558-e327-4178-a2ac-948e7f005ef1" (UID: "ded8f558-e327-4178-a2ac-948e7f005ef1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:34:27.360956 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.360898 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-socket" pod "ded8f558-e327-4178-a2ac-948e7f005ef1" (UID: "ded8f558-e327-4178-a2ac-948e7f005ef1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.361034 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.361013 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-credential-socket" pod "ded8f558-e327-4178-a2ac-948e7f005ef1" (UID: "ded8f558-e327-4178-a2ac-948e7f005ef1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.361094 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.361073 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.361155 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.361094 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded8f558-e327-4178-a2ac-948e7f005ef1-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.361155 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.361110 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.361264 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.361192 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-data" pod "ded8f558-e327-4178-a2ac-948e7f005ef1" (UID: "ded8f558-e327-4178-a2ac-948e7f005ef1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.362706 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.362660 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-podinfo" pod "ded8f558-e327-4178-a2ac-948e7f005ef1" (UID: "ded8f558-e327-4178-a2ac-948e7f005ef1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:34:27.363016 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.362998 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-kube-api-access-hf5m8" pod "ded8f558-e327-4178-a2ac-948e7f005ef1" (UID: "ded8f558-e327-4178-a2ac-948e7f005ef1"). InnerVolumeSpecName "kube-api-access-hf5m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:27.363291 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.363266 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-token" pod "ded8f558-e327-4178-a2ac-948e7f005ef1" (UID: "ded8f558-e327-4178-a2ac-948e7f005ef1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:27.363291 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.363271 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-envoy" pod "ded8f558-e327-4178-a2ac-948e7f005ef1" (UID: "ded8f558-e327-4178-a2ac-948e7f005ef1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.378183 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.378159 5015 generic.go:356] "Generic (PLEG): container finished" podID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" containerID="0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd" exitCode=0 Nov 03 10:34:27.378321 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.378206 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" Nov 03 10:34:27.378321 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.378244 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" event={"ID":"0d1bcdca-f1fe-45db-98df-66f3ebd5696e","Type":"ContainerDied","Data":"0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd"} Nov 03 10:34:27.378321 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.378305 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm" event={"ID":"0d1bcdca-f1fe-45db-98df-66f3ebd5696e","Type":"ContainerDied","Data":"fc9e4f4452b8e941ee13b38f7bc97b64512800eaab2e3fbe4c9362854b4729ed"} Nov 03 10:34:27.378539 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.378334 5015 scope.go:122] "RemoveContainer" containerID="0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd" Nov 03 10:34:27.379922 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.379894 5015 generic.go:356] "Generic (PLEG): container finished" podID="ded8f558-e327-4178-a2ac-948e7f005ef1" containerID="edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690" exitCode=0 Nov 03 10:34:27.380020 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.379922 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" event={"ID":"ded8f558-e327-4178-a2ac-948e7f005ef1","Type":"ContainerDied","Data":"edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690"} Nov 03 10:34:27.380020 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.379955 5015 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" Nov 03 10:34:27.380020 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.379967 5015 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc" event={"ID":"ded8f558-e327-4178-a2ac-948e7f005ef1","Type":"ContainerDied","Data":"61fa884327a0f4d80d50c74e32fbf76a81651ca08bcc00c4c9a73219eb2bbb3d"} Nov 03 10:34:27.395403 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.395384 5015 scope.go:122] "RemoveContainer" containerID="0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd" Nov 03 10:34:27.395763 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:34:27.395740 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd\": container with ID starting with 0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd not found: ID does not exist" containerID="0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd" Nov 03 10:34:27.395834 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.395773 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd"} err="failed to get container status \"0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd\": rpc error: code = NotFound desc = could not find container \"0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd\": container with ID starting with 0f8dac641ad56daf09118e9acc71d7ecd9262eecf016934b5170b0c991c33cbd not found: ID does not exist" Nov 03 10:34:27.395834 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.395795 5015 scope.go:122] "RemoveContainer" containerID="edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690" Nov 03 10:34:27.405401 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.405376 5015 scope.go:122] "RemoveContainer" containerID="edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690" Nov 03 10:34:27.405768 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:34:27.405752 5015 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690\": container with ID starting with edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690 not found: ID does not exist" containerID="edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690" Nov 03 10:34:27.405835 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.405778 5015 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690"} err="failed to get container status \"edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690\": rpc error: code = NotFound desc = could not find container \"edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690\": container with ID starting with edea21d41aa08a448578ba358941254cbdffaa5ea7a5da76309baa9e015eb690 not found: ID does not exist" Nov 03 10:34:27.462357 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.462324 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-data\") pod \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " Nov 03 10:34:27.462536 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.462419 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-token\" (UniqueName: \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-token\") pod \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " Nov 03 10:34:27.462536 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.462450 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-podinfo\") pod \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " Nov 03 10:34:27.462536 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.462487 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-socket\") pod \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " Nov 03 10:34:27.462749 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.462560 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-kube-api-access-bbwx5\" (UniqueName: \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-kube-api-access-bbwx5\") pod \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " Nov 03 10:34:27.462749 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.462593 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-certs\") pod \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " Nov 03 10:34:27.462749 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.462623 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-credential-socket\") pod \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " Nov 03 10:34:27.462749 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.462666 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istiod-ca-cert\") pod \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " Nov 03 10:34:27.462749 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.462695 5015 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-envoy\") pod \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\" (UID: \"0d1bcdca-f1fe-45db-98df-66f3ebd5696e\") " Nov 03 10:34:27.463191 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.463109 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-data" pod "0d1bcdca-f1fe-45db-98df-66f3ebd5696e" (UID: "0d1bcdca-f1fe-45db-98df-66f3ebd5696e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.463191 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.463165 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.463191 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.463187 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.463398 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.463199 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.463398 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.463211 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded8f558-e327-4178-a2ac-948e7f005ef1-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.463398 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.463223 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf5m8\" (UniqueName: \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-kube-api-access-hf5m8\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.463398 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.463225 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-credential-socket" pod "0d1bcdca-f1fe-45db-98df-66f3ebd5696e" (UID: "0d1bcdca-f1fe-45db-98df-66f3ebd5696e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.463398 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.463235 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded8f558-e327-4178-a2ac-948e7f005ef1-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.463867 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.463426 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istiod-ca-cert" pod "0d1bcdca-f1fe-45db-98df-66f3ebd5696e" (UID: "0d1bcdca-f1fe-45db-98df-66f3ebd5696e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:34:27.463867 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.463722 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-certs" pod "0d1bcdca-f1fe-45db-98df-66f3ebd5696e" (UID: "0d1bcdca-f1fe-45db-98df-66f3ebd5696e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.464141 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.464116 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-socket" pod "0d1bcdca-f1fe-45db-98df-66f3ebd5696e" (UID: "0d1bcdca-f1fe-45db-98df-66f3ebd5696e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.465679 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.465637 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-envoy" pod "0d1bcdca-f1fe-45db-98df-66f3ebd5696e" (UID: "0d1bcdca-f1fe-45db-98df-66f3ebd5696e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.465679 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.465646 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-kube-api-access-bbwx5" pod "0d1bcdca-f1fe-45db-98df-66f3ebd5696e" (UID: "0d1bcdca-f1fe-45db-98df-66f3ebd5696e"). InnerVolumeSpecName "kube-api-access-bbwx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:27.466125 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.466073 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-podinfo" pod "0d1bcdca-f1fe-45db-98df-66f3ebd5696e" (UID: "0d1bcdca-f1fe-45db-98df-66f3ebd5696e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:34:27.466590 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.466556 5015 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-token" pod "0d1bcdca-f1fe-45db-98df-66f3ebd5696e" (UID: "0d1bcdca-f1fe-45db-98df-66f3ebd5696e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:27.564699 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.564613 5015 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-token\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.564699 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.564653 5015 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-podinfo\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.564699 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.564670 5015 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.564699 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.564683 5015 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbwx5\" (UniqueName: \"kubernetes.io/projected/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-kube-api-access-bbwx5\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.564699 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.564701 5015 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-workload-certs\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.564982 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.564714 5015 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-credential-socket\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.564982 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.564727 5015 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istiod-ca-cert\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.564982 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.564740 5015 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-envoy\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.564982 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.564754 5015 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1bcdca-f1fe-45db-98df-66f3ebd5696e-istio-data\") on node \"ip-10-0-131-14.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.596949 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.596912 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc"] Nov 03 10:34:27.603674 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.603642 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-dhjmc"] Nov 03 10:34:27.837864 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.837774 5015 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm"] Nov 03 10:34:27.845655 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:27.845623 5015 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lg5gm"] Nov 03 10:34:28.464509 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:28.464447 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:34:28.486350 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:28.486326 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-b2jvc_a21ac76c-2765-417e-832a-32e140173e03/console-operator/3.log" Nov 03 10:34:29.166196 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:29.166168 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-qrhwd_6f925e33-6569-4ac4-9422-1ac38473de7a/main/0.log" Nov 03 10:34:29.204098 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:29.204061 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d1bcdca-f1fe-45db-98df-66f3ebd5696e" path="/var/lib/kubelet/pods/0d1bcdca-f1fe-45db-98df-66f3ebd5696e/volumes" Nov 03 10:34:29.204596 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:29.204581 5015 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded8f558-e327-4178-a2ac-948e7f005ef1" path="/var/lib/kubelet/pods/ded8f558-e327-4178-a2ac-948e7f005ef1/volumes" Nov 03 10:34:29.600772 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:29.600735 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8_12c4a382-3942-493e-84a4-85d061fc87e4/main/0.log" Nov 03 10:34:30.040749 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:30.040714 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9884s_4537f2eb-9880-4ba9-a64a-0c25e21521dd/main/0.log" Nov 03 10:34:30.259951 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:30.259917 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6_21ad0cfe-e3c6-4770-b0b3-7c512e7f1830/main/0.log" Nov 03 10:34:30.660600 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:30.660571 5015 scope.go:122] "RemoveContainer" containerID="10b77b15891cff5ec321f1081966e7caa3e4bf75d373e35323258632285f34c3" Nov 03 10:34:30.671842 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:30.671815 5015 scope.go:122] "RemoveContainer" containerID="11dc7381a2da4e5d2557c899415b9fd1e595d3d19b93bf97ded42c926dbf4f96" Nov 03 10:34:30.686964 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:30.686324 5015 scope.go:122] "RemoveContainer" containerID="46c3d3b251c36f5a2875446d8708d3caaa5d584fc0182a6562b4258833f2f3d4" Nov 03 10:34:30.699582 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:30.699559 5015 scope.go:122] "RemoveContainer" containerID="d9cc48f5bd5e04b94636ea1403b0b6903a915d1069b00262dcf5640ef2b96a3b" Nov 03 10:34:30.710581 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:30.710553 5015 scope.go:122] "RemoveContainer" containerID="8f6e18ea33bb44aa8c6a9d78e71f2141c45344155fe0a7ec72a401d34db0f5a2" Nov 03 10:34:46.919550 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:34:46.919505 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:34:48.303330 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:48.303273 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dfcc7c588-7hvqz_ae051d13-de71-4752-b918-8bb60b21002d/authorino/0.log" Nov 03 10:34:48.429609 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:48.429570 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rc42m_1786e9e7-2202-4fce-aaf4-2f0b30932688/manager/0.log" Nov 03 10:34:49.144144 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:34:49.144079 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cr2kg_19f7d098-9626-4724-a15e-85253b3dfd54/manager/0.log" Nov 03 10:35:06.415600 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:35:06.415569 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-qrhwd_6f925e33-6569-4ac4-9422-1ac38473de7a/main/0.log" Nov 03 10:35:06.852874 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:35:06.852833 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8_12c4a382-3942-493e-84a4-85d061fc87e4/main/0.log" Nov 03 10:35:07.286903 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:35:07.286867 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9884s_4537f2eb-9880-4ba9-a64a-0c25e21521dd/main/0.log" Nov 03 10:35:07.503879 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:35:07.503844 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6_21ad0cfe-e3c6-4770-b0b3-7c512e7f1830/main/0.log" Nov 03 10:35:16.911644 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:35:16.911602 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:35:35.288055 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:35:35.287955 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dfcc7c588-7hvqz_ae051d13-de71-4752-b918-8bb60b21002d/authorino/0.log" Nov 03 10:35:35.444530 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:35:35.442120 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rc42m_1786e9e7-2202-4fce-aaf4-2f0b30932688/manager/0.log" Nov 03 10:35:36.155381 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:35:36.155337 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cr2kg_19f7d098-9626-4724-a15e-85253b3dfd54/manager/0.log" Nov 03 10:35:46.906966 ip-10-0-131-14 kubenswrapper[5015]: E1103 10:35:46.906918 5015 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pc6m2" containerName="extractor" Nov 03 10:35:53.621711 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:35:53.621676 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-qrhwd_6f925e33-6569-4ac4-9422-1ac38473de7a/main/0.log" Nov 03 10:35:54.055205 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:35:54.055174 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m45k8_12c4a382-3942-493e-84a4-85d061fc87e4/main/0.log" Nov 03 10:35:54.488192 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:35:54.488161 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9884s_4537f2eb-9880-4ba9-a64a-0c25e21521dd/main/0.log" Nov 03 10:35:54.710465 ip-10-0-131-14 kubenswrapper[5015]: I1103 10:35:54.710430 5015 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ctbl6_21ad0cfe-e3c6-4770-b0b3-7c512e7f1830/main/0.log"